2026-02-22T20:16:59.776 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-22T20:16:59.791 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T20:16:59.872 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/nmordech-2026-02-22_19:44:01-rados-wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle-distro-default-trial/64480 branch: wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} email: null first_in_suite: false flavor: default job_id: '64480' kernel: branch: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: trial name: nmordech-2026-02-22_19:44:01-rados-wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a5c3084f688b1f6eff223b8d259a8316f9844c8e ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: distro-suffix: rockylinux-10 install: ceph: flavor: default sha1: a5c3084f688b1f6eff223b8d259a8316f9844c8e workunit: branch: wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle sha1: a5c3084f688b1f6eff223b8d259a8316f9844c8e owner: scheduled_nmordech@soko04.front.sepia.ceph.com priority: 10 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 966 sha1: a5c3084f688b1f6eff223b8d259a8316f9844c8e sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_a5c3084f688b1f6eff223b8d259a8316f9844c8e/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: a5c3084f688b1f6eff223b8d259a8316f9844c8e targets: trial150.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAASsSbCEo2cK+67HKz2MECnvdwyRlKGVysW/FfwUYcm91zUGrOj2TBdpmaeIQNorP4l4JgVCmd5+SxmdeclVR8= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-22_19:44:01 tube: trial user: nmordech verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3318653 2026-02-22T20:16:59.872 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_a5c3084f688b1f6eff223b8d259a8316f9844c8e/qa; will attempt to use it 2026-02-22T20:16:59.872 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_a5c3084f688b1f6eff223b8d259a8316f9844c8e/qa/tasks 2026-02-22T20:16:59.872 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-22T20:16:59.874 INFO:teuthology.task.internal:Checking packages... 2026-02-22T20:16:59.874 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'a5c3084f688b1f6eff223b8d259a8316f9844c8e' 2026-02-22T20:16:59.874 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-22T20:16:59.874 INFO:teuthology.packaging:ref: None 2026-02-22T20:16:59.874 INFO:teuthology.packaging:tag: None 2026-02-22T20:16:59.874 INFO:teuthology.packaging:branch: wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle 2026-02-22T20:16:59.874 INFO:teuthology.packaging:sha1: a5c3084f688b1f6eff223b8d259a8316f9844c8e 2026-02-22T20:16:59.875 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle 2026-02-22T20:17:00.002 INFO:teuthology.task.internal:Found packages for ceph version 20.2.0-794-ga5c3084f-1jammy 2026-02-22T20:17:00.003 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-22T20:17:00.014 INFO:teuthology.task.internal:no buildpackages task found 2026-02-22T20:17:00.014 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-22T20:17:00.030 INFO:teuthology.task.internal:Saving configuration 2026-02-22T20:17:00.044 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-22T20:17:00.055 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-22T20:17:00.134 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial150.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/nmordech-2026-02-22_19:44:01-rados-wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle-distro-default-trial/64480', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-22 20:13:28.599128', 'locked_by': 'scheduled_nmordech@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID4sDMKrD6pqo7fwOR7lBtP1T2+bOKLAcqRYw2o99J61'} 2026-02-22T20:17:00.134 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-22T20:17:00.144 INFO:teuthology.task.internal:roles: ubuntu@trial150.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-22T20:17:00.144 INFO:teuthology.run_tasks:Running task console_log... 2026-02-22T20:17:00.232 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7b9e869213a0>, signals=[15]) 2026-02-22T20:17:00.232 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-22T20:17:00.246 INFO:teuthology.task.internal:Opening connections... 2026-02-22T20:17:00.246 DEBUG:teuthology.task.internal:connecting to ubuntu@trial150.front.sepia.ceph.com 2026-02-22T20:17:00.247 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T20:17:00.309 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-22T20:17:00.321 DEBUG:teuthology.orchestra.run.trial150:> uname -m 2026-02-22T20:17:00.387 INFO:teuthology.orchestra.run.trial150.stdout:x86_64 2026-02-22T20:17:00.387 DEBUG:teuthology.orchestra.run.trial150:> cat /etc/os-release 2026-02-22T20:17:00.432 INFO:teuthology.orchestra.run.trial150.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-22T20:17:00.432 INFO:teuthology.orchestra.run.trial150.stdout:NAME="Ubuntu" 2026-02-22T20:17:00.433 INFO:teuthology.orchestra.run.trial150.stdout:VERSION_ID="22.04" 2026-02-22T20:17:00.433 INFO:teuthology.orchestra.run.trial150.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-22T20:17:00.433 INFO:teuthology.orchestra.run.trial150.stdout:VERSION_CODENAME=jammy 2026-02-22T20:17:00.433 INFO:teuthology.orchestra.run.trial150.stdout:ID=ubuntu 2026-02-22T20:17:00.433 INFO:teuthology.orchestra.run.trial150.stdout:ID_LIKE=debian 2026-02-22T20:17:00.433 INFO:teuthology.orchestra.run.trial150.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-22T20:17:00.433 INFO:teuthology.orchestra.run.trial150.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-22T20:17:00.433 INFO:teuthology.orchestra.run.trial150.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-22T20:17:00.433 INFO:teuthology.orchestra.run.trial150.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-22T20:17:00.433 INFO:teuthology.orchestra.run.trial150.stdout:UBUNTU_CODENAME=jammy 2026-02-22T20:17:00.433 INFO:teuthology.lock.ops:Updating trial150.front.sepia.ceph.com on lock server 2026-02-22T20:17:00.493 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-22T20:17:00.505 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-22T20:17:00.517 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-22T20:17:00.517 DEBUG:teuthology.orchestra.run.trial150:> test '!' -e /home/ubuntu/cephtest 2026-02-22T20:17:00.520 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-22T20:17:00.531 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-22T20:17:00.531 DEBUG:teuthology.orchestra.run.trial150:> test -z $(ls -A /var/lib/ceph) 2026-02-22T20:17:00.565 INFO:teuthology.orchestra.run.trial150.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-22T20:17:00.566 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-22T20:17:00.651 INFO:teuthology.run_tasks:Running task kernel... 2026-02-22T20:17:00.671 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-22T20:17:00.671 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-22T20:17:00.671 DEBUG:teuthology.orchestra.run.trial150:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-22T20:17:00.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T20:17:00.675 DEBUG:teuthology.orchestra.run.trial150:> uname -r 2026-02-22T20:17:00.721 INFO:teuthology.orchestra.run.trial150.stdout:5.15.0-170-generic 2026-02-22T20:17:00.721 INFO:teuthology.task.kernel:Running kernel on trial150: 5.15.0-170-generic 2026-02-22T20:17:00.722 DEBUG:teuthology.orchestra.run.trial150:> sudo apt-get clean 2026-02-22T20:17:00.780 DEBUG:teuthology.orchestra.run.trial150:> sudo apt-get update 2026-02-22T20:17:00.961 INFO:teuthology.orchestra.run.trial150.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T20:17:00.968 INFO:teuthology.orchestra.run.trial150.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-22T20:17:01.008 INFO:teuthology.orchestra.run.trial150.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-22T20:17:01.023 INFO:teuthology.orchestra.run.trial150.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-22T20:17:01.104 INFO:teuthology.orchestra.run.trial150.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-22T20:17:01.168 INFO:teuthology.orchestra.run.trial150.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-22T20:17:01.172 INFO:teuthology.orchestra.run.trial150.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-22T20:17:01.172 INFO:teuthology.orchestra.run.trial150.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-22T20:17:01.181 INFO:teuthology.orchestra.run.trial150.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-22T20:17:01.183 INFO:teuthology.orchestra.run.trial150.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-22T20:17:01.183 INFO:teuthology.orchestra.run.trial150.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-22T20:17:01.209 INFO:teuthology.orchestra.run.trial150.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-22T20:17:01.212 INFO:teuthology.orchestra.run.trial150.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-22T20:17:01.222 INFO:teuthology.orchestra.run.trial150.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-22T20:17:01.223 INFO:teuthology.orchestra.run.trial150.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-22T20:17:01.223 INFO:teuthology.orchestra.run.trial150.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-22T20:17:01.227 INFO:teuthology.orchestra.run.trial150.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-22T20:17:01.227 INFO:teuthology.orchestra.run.trial150.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-22T20:17:01.244 INFO:teuthology.orchestra.run.trial150.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-22T20:17:05.078 INFO:teuthology.orchestra.run.trial150.stdout:Fetched 22.6 MB in 2s (13.7 MB/s) 2026-02-22T20:17:05.603 INFO:teuthology.orchestra.run.trial150.stdout:Reading package lists... 2026-02-22T20:17:05.617 DEBUG:teuthology.orchestra.run.trial150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-22T20:17:05.650 INFO:teuthology.orchestra.run.trial150.stdout:Reading package lists... 2026-02-22T20:17:05.756 INFO:teuthology.orchestra.run.trial150.stdout:Building dependency tree... 2026-02-22T20:17:05.757 INFO:teuthology.orchestra.run.trial150.stdout:Reading state information... 2026-02-22T20:17:05.842 INFO:teuthology.orchestra.run.trial150.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-22T20:17:05.842 INFO:teuthology.orchestra.run.trial150.stdout:linux-image-generic set to manually installed. 2026-02-22T20:17:05.842 INFO:teuthology.orchestra.run.trial150.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T20:17:05.842 INFO:teuthology.orchestra.run.trial150.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T20:17:05.842 INFO:teuthology.orchestra.run.trial150.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T20:17:05.842 INFO:teuthology.orchestra.run.trial150.stdout: sg3-utils-udev 2026-02-22T20:17:05.842 INFO:teuthology.orchestra.run.trial150.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T20:17:05.861 INFO:teuthology.orchestra.run.trial150.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T20:17:05.863 DEBUG:teuthology.orchestra.run.trial150:> dpkg -s linux-image-generic 2026-02-22T20:17:05.916 INFO:teuthology.orchestra.run.trial150.stdout:Package: linux-image-generic 2026-02-22T20:17:05.916 INFO:teuthology.orchestra.run.trial150.stdout:Status: install ok installed 2026-02-22T20:17:05.916 INFO:teuthology.orchestra.run.trial150.stdout:Priority: optional 2026-02-22T20:17:05.916 INFO:teuthology.orchestra.run.trial150.stdout:Section: kernel 2026-02-22T20:17:05.916 INFO:teuthology.orchestra.run.trial150.stdout:Installed-Size: 21 2026-02-22T20:17:05.916 INFO:teuthology.orchestra.run.trial150.stdout:Maintainer: Ubuntu Kernel Team 2026-02-22T20:17:05.916 INFO:teuthology.orchestra.run.trial150.stdout:Architecture: amd64 2026-02-22T20:17:05.916 INFO:teuthology.orchestra.run.trial150.stdout:Source: linux-meta 2026-02-22T20:17:05.917 INFO:teuthology.orchestra.run.trial150.stdout:Version: 5.15.0.170.159 2026-02-22T20:17:05.917 INFO:teuthology.orchestra.run.trial150.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-22T20:17:05.917 INFO:teuthology.orchestra.run.trial150.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-22T20:17:05.917 INFO:teuthology.orchestra.run.trial150.stdout:Recommends: thermald 2026-02-22T20:17:05.917 INFO:teuthology.orchestra.run.trial150.stdout:Description: Generic Linux kernel image 2026-02-22T20:17:05.917 INFO:teuthology.orchestra.run.trial150.stdout: This package will always depend on the latest generic kernel image 2026-02-22T20:17:05.917 INFO:teuthology.orchestra.run.trial150.stdout: available. 2026-02-22T20:17:05.917 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-22T20:17:05.918 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-22T20:17:05.918 DEBUG:teuthology.orchestra.run.trial150:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-22T20:17:05.966 INFO:teuthology.orchestra.run.trial150.stdout:ttyS1 2026-02-22T20:17:05.988 DEBUG:teuthology.parallel:result is None 2026-02-22T20:17:05.988 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-22T20:17:06.006 INFO:teuthology.task.internal:Creating test directory... 2026-02-22T20:17:06.006 DEBUG:teuthology.orchestra.run.trial150:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-22T20:17:06.034 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-22T20:17:06.045 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-22T20:17:06.062 INFO:teuthology.task.internal:Creating archive directory... 2026-02-22T20:17:06.063 DEBUG:teuthology.orchestra.run.trial150:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-22T20:17:06.082 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-22T20:17:06.094 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-22T20:17:06.094 DEBUG:teuthology.orchestra.run.trial150:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-22T20:17:06.130 INFO:teuthology.orchestra.run.trial150.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:17:06.137 INFO:teuthology.orchestra.run.trial150.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:17:06.138 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-22T20:17:06.150 INFO:teuthology.task.internal:Configuring sudo... 2026-02-22T20:17:06.150 DEBUG:teuthology.orchestra.run.trial150:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-22T20:17:06.190 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-22T20:17:06.208 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-22T20:17:06.208 DEBUG:teuthology.orchestra.run.trial150:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-22T20:17:06.241 DEBUG:teuthology.orchestra.run.trial150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T20:17:06.286 DEBUG:teuthology.orchestra.run.trial150:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T20:17:06.333 DEBUG:teuthology.orchestra.run.trial150:> set -ex 2026-02-22T20:17:06.334 DEBUG:teuthology.orchestra.run.trial150:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-22T20:17:06.383 DEBUG:teuthology.orchestra.run.trial150:> sudo service rsyslog restart 2026-02-22T20:17:06.443 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-22T20:17:06.456 INFO:teuthology.task.internal:Starting timer... 2026-02-22T20:17:06.456 INFO:teuthology.run_tasks:Running task pcp... 2026-02-22T20:17:06.481 INFO:teuthology.run_tasks:Running task selinux... 2026-02-22T20:17:06.494 DEBUG:teuthology.task.selinux:Excluding trial150: OS 'ubuntu' does not support SELinux 2026-02-22T20:17:06.494 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-22T20:17:06.494 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-22T20:17:06.494 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-22T20:17:06.494 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-22T20:17:06.512 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-22T20:17:06.522 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-22T20:17:06.524 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial150.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-22T20:19:24.981 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial150.front.sepia.ceph.com')] 2026-02-22T20:19:24.989 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial150.front.sepia.ceph.com' 2026-02-22T20:19:24.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial150.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-22T20:19:25.098 DEBUG:teuthology.orchestra.run.trial150:> true 2026-02-22T20:19:25.347 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial150.front.sepia.ceph.com' 2026-02-22T20:19:25.347 INFO:teuthology.run_tasks:Running task clock... 2026-02-22T20:19:25.363 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-22T20:19:25.364 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-22T20:19:25.365 DEBUG:teuthology.orchestra.run.trial150:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T20:19:25.404 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-22T20:19:25.404 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: Command line: ntpd -gq 2026-02-22T20:19:25.404 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: ---------------------------------------------------- 2026-02-22T20:19:25.404 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: ntp-4 is maintained by Network Time Foundation, 2026-02-22T20:19:25.405 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-22T20:19:25.405 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: corporation. Support and training for ntp-4 are 2026-02-22T20:19:25.405 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: available at https://www.nwtime.org/support 2026-02-22T20:19:25.405 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: ---------------------------------------------------- 2026-02-22T20:19:25.405 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: proto: precision = 0.020 usec (-25) 2026-02-22T20:19:25.405 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: basedate set to 2022-02-04 2026-02-22T20:19:25.405 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: gps base set to 2022-02-06 (week 2196) 2026-02-22T20:19:25.405 INFO:teuthology.orchestra.run.trial150.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T20:19:25.406 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-22T20:19:25.406 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: restrict ::: KOD does nothing without LIMITED. 2026-02-22T20:19:25.406 INFO:teuthology.orchestra.run.trial150.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-22T20:19:25.406 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: Listen and drop on 0 v6wildcard [::]:123 2026-02-22T20:19:25.406 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-22T20:19:25.406 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: Listen normally on 2 lo 127.0.0.1:123 2026-02-22T20:19:25.406 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: Listen normally on 3 enp1s0f0 10.20.193.150:123 2026-02-22T20:19:25.407 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: Listen normally on 4 lo [::1]:123 2026-02-22T20:19:25.407 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:6494%2]:123 2026-02-22T20:19:25.407 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:25 ntpd[9430]: Listening on routing socket on fd #22 for interface updates 2026-02-22T20:19:26.405 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:26 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:26.405 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:26 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:26.405 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:26 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:26.406 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:26 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:26.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:26 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:26.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:26 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:26.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:26 ntpd[9430]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T20:19:26.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:26 ntpd[9430]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T20:19:28.405 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:28.405 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:28.406 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:28.406 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:28.406 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:28.406 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:28.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:28.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:28.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T20:19:28.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T20:19:28.413 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:28.413 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:28.413 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T20:19:28.413 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T20:19:28.478 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:28.478 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:28.478 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T20:19:28.479 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:28 ntpd[9430]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T20:19:30.405 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:30.405 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:30.405 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:30.406 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:30.406 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:30.406 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:30.408 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:30.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:30.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T20:19:30.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T20:19:30.437 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:30.438 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:30.438 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T20:19:30.438 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T20:19:30.474 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:30.474 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:30.475 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T20:19:30.475 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:30 ntpd[9430]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T20:19:32.405 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:32 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:32.405 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:32 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:32.405 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:32 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:32.406 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:32 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:32.406 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:32 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:32.406 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:32 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:32.408 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:32 ntpd[9430]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-22T20:19:32.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:32 ntpd[9430]: can't open /var/log/ntpstats/rawstats.20260222: Permission denied 2026-02-22T20:19:32.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:32 ntpd[9430]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-22T20:19:32.409 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:32 ntpd[9430]: can't open /var/log/ntpstats/peerstats.20260222: Permission denied 2026-02-22T20:19:32.409 INFO:teuthology.orchestra.run.trial150.stdout:22 Feb 20:19:32 ntpd[9430]: ntpd: time set -0.092379 s 2026-02-22T20:19:32.409 INFO:teuthology.orchestra.run.trial150.stdout:ntpd: time set -0.092379s 2026-02-22T20:19:32.410 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:32 ntpd[9430]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-22T20:19:32.410 INFO:teuthology.orchestra.run.trial150.stderr:22 Feb 20:19:32 ntpd[9430]: can't open /var/log/ntpstats/loopstats.20260222: Permission denied 2026-02-22T20:19:32.452 INFO:teuthology.orchestra.run.trial150.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T20:19:32.452 INFO:teuthology.orchestra.run.trial150.stdout:============================================================================== 2026-02-22T20:19:32.452 INFO:teuthology.orchestra.run.trial150.stdout: 172-233-177-198 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T20:19:32.452 INFO:teuthology.orchestra.run.trial150.stdout: rn-02.koehn.com .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-22T20:19:32.453 INFO:teuthology.run_tasks:Running task install... 2026-02-22T20:19:32.467 DEBUG:teuthology.task.install:project ceph 2026-02-22T20:19:32.468 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a5c3084f688b1f6eff223b8d259a8316f9844c8e'}} 2026-02-22T20:19:32.468 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'a5c3084f688b1f6eff223b8d259a8316f9844c8e'} 2026-02-22T20:19:32.468 INFO:teuthology.task.install:Using flavor: default 2026-02-22T20:19:32.478 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-22T20:19:32.479 INFO:teuthology.task.install:extra packages: [] 2026-02-22T20:19:32.479 DEBUG:teuthology.orchestra.run.trial150:> sudo apt-key list | grep Ceph 2026-02-22T20:19:32.546 INFO:teuthology.orchestra.run.trial150.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-22T20:19:32.573 INFO:teuthology.orchestra.run.trial150.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-22T20:19:32.573 INFO:teuthology.orchestra.run.trial150.stdout:uid [ unknown] Ceph.com (release key) 2026-02-22T20:19:32.575 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-22T20:19:32.575 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a5c3084f688b1f6eff223b8d259a8316f9844c8e 2026-02-22T20:19:32.703 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default/ 2026-02-22T20:19:32.703 INFO:teuthology.task.install.deb:Package version is 20.2.0-794-ga5c3084f-1jammy 2026-02-22T20:19:32.846 DEBUG:teuthology.orchestra.run.trial150:> set -ex 2026-02-22T20:19:32.846 DEBUG:teuthology.orchestra.run.trial150:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-22T20:19:32.856 DEBUG:teuthology.orchestra.run.trial150:> sudo apt-get update 2026-02-22T20:19:33.020 INFO:teuthology.orchestra.run.trial150.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy InRelease 2026-02-22T20:19:33.036 INFO:teuthology.orchestra.run.trial150.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy Release [7680 B] 2026-02-22T20:19:33.051 INFO:teuthology.orchestra.run.trial150.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-22T20:19:33.066 INFO:teuthology.orchestra.run.trial150.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.9 kB] 2026-02-22T20:19:33.299 INFO:teuthology.orchestra.run.trial150.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-22T20:19:33.365 INFO:teuthology.orchestra.run.trial150.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-22T20:19:33.438 INFO:teuthology.orchestra.run.trial150.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-22T20:19:33.510 INFO:teuthology.orchestra.run.trial150.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-22T20:19:33.657 INFO:teuthology.orchestra.run.trial150.stdout:Fetched 26.5 kB in 1s (41.8 kB/s) 2026-02-22T20:19:34.185 INFO:teuthology.orchestra.run.trial150.stdout:Reading package lists... 2026-02-22T20:19:34.199 DEBUG:teuthology.orchestra.run.trial150:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.2.0-794-ga5c3084f-1jammy cephadm=20.2.0-794-ga5c3084f-1jammy ceph-mds=20.2.0-794-ga5c3084f-1jammy ceph-mgr=20.2.0-794-ga5c3084f-1jammy ceph-common=20.2.0-794-ga5c3084f-1jammy ceph-fuse=20.2.0-794-ga5c3084f-1jammy ceph-test=20.2.0-794-ga5c3084f-1jammy ceph-volume=20.2.0-794-ga5c3084f-1jammy radosgw=20.2.0-794-ga5c3084f-1jammy python3-rados=20.2.0-794-ga5c3084f-1jammy python3-rgw=20.2.0-794-ga5c3084f-1jammy python3-cephfs=20.2.0-794-ga5c3084f-1jammy python3-rbd=20.2.0-794-ga5c3084f-1jammy libcephfs2=20.2.0-794-ga5c3084f-1jammy libcephfs-dev=20.2.0-794-ga5c3084f-1jammy librados2=20.2.0-794-ga5c3084f-1jammy librbd1=20.2.0-794-ga5c3084f-1jammy rbd-fuse=20.2.0-794-ga5c3084f-1jammy 2026-02-22T20:19:34.232 INFO:teuthology.orchestra.run.trial150.stdout:Reading package lists... 2026-02-22T20:19:34.341 INFO:teuthology.orchestra.run.trial150.stdout:Building dependency tree... 2026-02-22T20:19:34.341 INFO:teuthology.orchestra.run.trial150.stdout:Reading state information... 2026-02-22T20:19:34.432 INFO:teuthology.orchestra.run.trial150.stdout:The following packages were automatically installed and are no longer required: 2026-02-22T20:19:34.432 INFO:teuthology.orchestra.run.trial150.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: sg3-utils-udev 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout:The following additional packages will be installed: 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: libcephfs-daemon libcephfs-proxy2 libdouble-conversion3 libfuse2 libjq1 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: liblttng-ust1 libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: libqt5dbus5 libqt5network5 libradosstriper1 librdkafka1 librgw2 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli python-asyncssh-doc 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: python3-asyncssh python3-cachetools python3-ceph-argparse 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: python3-ceph-common python3-cheroot python3-cherrypy3 python3-google-auth 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: python3-jaraco.functools python3-jaraco.text python3-joblib 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: python3-kubernetes python3-natsort python3-pluggy python3-portend 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2026-02-22T20:19:34.433 INFO:teuthology.orchestra.run.trial150.stdout: python3-rsa python3-simplejson python3-sklearn python3-sklearn-lib 2026-02-22T20:19:34.434 INFO:teuthology.orchestra.run.trial150.stdout: python3-tempora python3-threadpoolctl python3-toml python3-wcwidth 2026-02-22T20:19:34.434 INFO:teuthology.orchestra.run.trial150.stdout: python3-webob python3-websocket python3-zc.lockfile qttranslations5-l10n 2026-02-22T20:19:34.434 INFO:teuthology.orchestra.run.trial150.stdout: socat xmlstarlet 2026-02-22T20:19:34.434 INFO:teuthology.orchestra.run.trial150.stdout:Suggested packages: 2026-02-22T20:19:34.434 INFO:teuthology.orchestra.run.trial150.stdout: python3-influxdb liblua5.3-dev luarocks python-natsort-doc python-psutil-doc 2026-02-22T20:19:34.434 INFO:teuthology.orchestra.run.trial150.stdout: subversion python-pygments-doc ttf-bitstream-vera python3-paste python3-dap 2026-02-22T20:19:34.434 INFO:teuthology.orchestra.run.trial150.stdout: python-sklearn-doc ipython3 python-webob-doc 2026-02-22T20:19:34.434 INFO:teuthology.orchestra.run.trial150.stdout:Recommended packages: 2026-02-22T20:19:34.434 INFO:teuthology.orchestra.run.trial150.stdout: btrfs-tools 2026-02-22T20:19:34.466 INFO:teuthology.orchestra.run.trial150.stdout:The following NEW packages will be installed: 2026-02-22T20:19:34.466 INFO:teuthology.orchestra.run.trial150.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-22T20:19:34.466 INFO:teuthology.orchestra.run.trial150.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: libcephfs-daemon libcephfs-dev libcephfs-proxy2 libcephfs2 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 liboath0 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python3-asyncssh 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-natsort 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: python3-pygments python3-pytest python3-rados python3-rbd python3-repoze.lru 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: python3-threadpoolctl python3-toml python3-wcwidth python3-webob 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: python3-websocket python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2026-02-22T20:19:34.467 INFO:teuthology.orchestra.run.trial150.stdout: socat xmlstarlet 2026-02-22T20:19:34.545 INFO:teuthology.orchestra.run.trial150.stdout:0 upgraded, 86 newly installed, 0 to remove and 13 not upgraded. 2026-02-22T20:19:34.545 INFO:teuthology.orchestra.run.trial150.stdout:Need to get 280 MB of archives. 2026-02-22T20:19:34.545 INFO:teuthology.orchestra.run.trial150.stdout:After this operation, 1119 MB of additional disk space will be used. 2026-02-22T20:19:34.545 INFO:teuthology.orchestra.run.trial150.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 20.2.0-794-ga5c3084f-1jammy [3569 kB] 2026-02-22T20:19:34.571 INFO:teuthology.orchestra.run.trial150.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-22T20:19:34.616 INFO:teuthology.orchestra.run.trial150.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-22T20:19:34.618 INFO:teuthology.orchestra.run.trial150.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-22T20:19:34.627 INFO:teuthology.orchestra.run.trial150.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-22T20:19:34.648 INFO:teuthology.orchestra.run.trial150.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 20.2.0-794-ga5c3084f-1jammy [2867 kB] 2026-02-22T20:19:34.656 INFO:teuthology.orchestra.run.trial150.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-22T20:19:34.658 INFO:teuthology.orchestra.run.trial150.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-22T20:19:34.663 INFO:teuthology.orchestra.run.trial150.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-22T20:19:34.665 INFO:teuthology.orchestra.run.trial150.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-22T20:19:34.665 INFO:teuthology.orchestra.run.trial150.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-22T20:19:34.666 INFO:teuthology.orchestra.run.trial150.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-22T20:19:34.666 INFO:teuthology.orchestra.run.trial150.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-22T20:19:34.671 INFO:teuthology.orchestra.run.trial150.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-22T20:19:34.671 INFO:teuthology.orchestra.run.trial150.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-22T20:19:34.671 INFO:teuthology.orchestra.run.trial150.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-22T20:19:34.677 INFO:teuthology.orchestra.run.trial150.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-22T20:19:34.677 INFO:teuthology.orchestra.run.trial150.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-22T20:19:34.677 INFO:teuthology.orchestra.run.trial150.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-22T20:19:34.677 INFO:teuthology.orchestra.run.trial150.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-22T20:19:34.686 INFO:teuthology.orchestra.run.trial150.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-22T20:19:34.686 INFO:teuthology.orchestra.run.trial150.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-22T20:19:34.687 INFO:teuthology.orchestra.run.trial150.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-22T20:19:34.687 INFO:teuthology.orchestra.run.trial150.stdout:Get:24 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 20.2.0-794-ga5c3084f-1jammy [833 kB] 2026-02-22T20:19:34.688 INFO:teuthology.orchestra.run.trial150.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-22T20:19:34.688 INFO:teuthology.orchestra.run.trial150.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-22T20:19:34.689 INFO:teuthology.orchestra.run.trial150.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-22T20:19:34.695 INFO:teuthology.orchestra.run.trial150.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-22T20:19:34.696 INFO:teuthology.orchestra.run.trial150.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-22T20:19:34.696 INFO:teuthology.orchestra.run.trial150.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-22T20:19:34.700 INFO:teuthology.orchestra.run.trial150.stdout:Get:31 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 20.2.0-794-ga5c3084f-1jammy [364 kB] 2026-02-22T20:19:34.705 INFO:teuthology.orchestra.run.trial150.stdout:Get:32 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 20.2.0-794-ga5c3084f-1jammy [32.8 kB] 2026-02-22T20:19:34.706 INFO:teuthology.orchestra.run.trial150.stdout:Get:33 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 20.2.0-794-ga5c3084f-1jammy [184 kB] 2026-02-22T20:19:34.709 INFO:teuthology.orchestra.run.trial150.stdout:Get:34 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 20.2.0-794-ga5c3084f-1jammy [87.0 kB] 2026-02-22T20:19:34.710 INFO:teuthology.orchestra.run.trial150.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-22T20:19:34.711 INFO:teuthology.orchestra.run.trial150.stdout:Get:36 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 20.2.0-794-ga5c3084f-1jammy [341 kB] 2026-02-22T20:19:34.712 INFO:teuthology.orchestra.run.trial150.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-22T20:19:34.712 INFO:teuthology.orchestra.run.trial150.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-22T20:19:34.716 INFO:teuthology.orchestra.run.trial150.stdout:Get:39 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 20.2.0-794-ga5c3084f-1jammy [8698 kB] 2026-02-22T20:19:34.723 INFO:teuthology.orchestra.run.trial150.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-22T20:19:34.723 INFO:teuthology.orchestra.run.trial150.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-22T20:19:34.724 INFO:teuthology.orchestra.run.trial150.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-22T20:19:34.724 INFO:teuthology.orchestra.run.trial150.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-22T20:19:34.724 INFO:teuthology.orchestra.run.trial150.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-22T20:19:34.724 INFO:teuthology.orchestra.run.trial150.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-22T20:19:34.726 INFO:teuthology.orchestra.run.trial150.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-22T20:19:34.726 INFO:teuthology.orchestra.run.trial150.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-22T20:19:34.727 INFO:teuthology.orchestra.run.trial150.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-22T20:19:34.733 INFO:teuthology.orchestra.run.trial150.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-22T20:19:34.737 INFO:teuthology.orchestra.run.trial150.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-22T20:19:34.739 INFO:teuthology.orchestra.run.trial150.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-22T20:19:34.741 INFO:teuthology.orchestra.run.trial150.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-22T20:19:34.743 INFO:teuthology.orchestra.run.trial150.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-22T20:19:34.743 INFO:teuthology.orchestra.run.trial150.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-22T20:19:34.743 INFO:teuthology.orchestra.run.trial150.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-22T20:19:34.744 INFO:teuthology.orchestra.run.trial150.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-22T20:19:34.744 INFO:teuthology.orchestra.run.trial150.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-22T20:19:34.747 INFO:teuthology.orchestra.run.trial150.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-22T20:19:34.752 INFO:teuthology.orchestra.run.trial150.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-22T20:19:34.753 INFO:teuthology.orchestra.run.trial150.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-22T20:19:34.761 INFO:teuthology.orchestra.run.trial150.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-22T20:19:34.762 INFO:teuthology.orchestra.run.trial150.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-22T20:19:34.859 INFO:teuthology.orchestra.run.trial150.stdout:Get:63 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 20.2.0-794-ga5c3084f-1jammy [112 kB] 2026-02-22T20:19:34.860 INFO:teuthology.orchestra.run.trial150.stdout:Get:64 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 20.2.0-794-ga5c3084f-1jammy [261 kB] 2026-02-22T20:19:34.865 INFO:teuthology.orchestra.run.trial150.stdout:Get:65 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 20.2.0-794-ga5c3084f-1jammy [29.3 MB] 2026-02-22T20:19:35.350 INFO:teuthology.orchestra.run.trial150.stdout:Get:66 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 20.2.0-794-ga5c3084f-1jammy [5416 kB] 2026-02-22T20:19:35.434 INFO:teuthology.orchestra.run.trial150.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 20.2.0-794-ga5c3084f-1jammy [250 kB] 2026-02-22T20:19:35.437 INFO:teuthology.orchestra.run.trial150.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 20.2.0-794-ga5c3084f-1jammy [124 kB] 2026-02-22T20:19:35.440 INFO:teuthology.orchestra.run.trial150.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 20.2.0-794-ga5c3084f-1jammy [910 kB] 2026-02-22T20:19:35.454 INFO:teuthology.orchestra.run.trial150.stdout:Get:70 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 20.2.0-794-ga5c3084f-1jammy [6401 kB] 2026-02-22T20:19:35.560 INFO:teuthology.orchestra.run.trial150.stdout:Get:71 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 20.2.0-794-ga5c3084f-1jammy [21.7 MB] 2026-02-22T20:19:35.918 INFO:teuthology.orchestra.run.trial150.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 20.2.0-794-ga5c3084f-1jammy [14.1 kB] 2026-02-22T20:19:35.919 INFO:teuthology.orchestra.run.trial150.stdout:Get:73 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 20.2.0-794-ga5c3084f-1jammy [957 kB] 2026-02-22T20:19:35.930 INFO:teuthology.orchestra.run.trial150.stdout:Get:74 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 20.2.0-794-ga5c3084f-1jammy [2344 kB] 2026-02-22T20:19:35.970 INFO:teuthology.orchestra.run.trial150.stdout:Get:75 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 20.2.0-794-ga5c3084f-1jammy [1057 kB] 2026-02-22T20:19:35.987 INFO:teuthology.orchestra.run.trial150.stdout:Get:76 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 20.2.0-794-ga5c3084f-1jammy [185 kB] 2026-02-22T20:19:35.990 INFO:teuthology.orchestra.run.trial150.stdout:Get:77 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 20.2.0-794-ga5c3084f-1jammy [45.5 MB] 2026-02-22T20:19:36.741 INFO:teuthology.orchestra.run.trial150.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 20.2.0-794-ga5c3084f-1jammy [8626 kB] 2026-02-22T20:19:36.877 INFO:teuthology.orchestra.run.trial150.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 20.2.0-794-ga5c3084f-1jammy [14.2 kB] 2026-02-22T20:19:36.877 INFO:teuthology.orchestra.run.trial150.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 20.2.0-794-ga5c3084f-1jammy [99.5 MB] 2026-02-22T20:19:38.522 INFO:teuthology.orchestra.run.trial150.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 20.2.0-794-ga5c3084f-1jammy [136 kB] 2026-02-22T20:19:38.522 INFO:teuthology.orchestra.run.trial150.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-daemon amd64 20.2.0-794-ga5c3084f-1jammy [43.3 kB] 2026-02-22T20:19:38.523 INFO:teuthology.orchestra.run.trial150.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-proxy2 amd64 20.2.0-794-ga5c3084f-1jammy [30.7 kB] 2026-02-22T20:19:38.523 INFO:teuthology.orchestra.run.trial150.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 20.2.0-794-ga5c3084f-1jammy [41.5 kB] 2026-02-22T20:19:38.523 INFO:teuthology.orchestra.run.trial150.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 20.2.0-794-ga5c3084f-1jammy [25.1 MB] 2026-02-22T20:19:38.923 INFO:teuthology.orchestra.run.trial150.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle/a5c3084f688b1f6eff223b8d259a8316f9844c8e/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 20.2.0-794-ga5c3084f-1jammy [97.5 kB] 2026-02-22T20:19:39.210 INFO:teuthology.orchestra.run.trial150.stdout:Fetched 280 MB in 4s (62.9 MB/s) 2026-02-22T20:19:39.227 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-22T20:19:39.382 INFO:teuthology.orchestra.run.trial150.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-22T20:19:39.384 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../00-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-22T20:19:39.386 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T20:19:39.408 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-22T20:19:39.420 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../01-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-22T20:19:39.420 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T20:19:39.437 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-22T20:19:39.449 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../02-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-22T20:19:39.449 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T20:19:39.471 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-22T20:19:39.479 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../03-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T20:19:39.483 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T20:19:39.517 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-22T20:19:39.526 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../04-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T20:19:39.527 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T20:19:39.543 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-22T20:19:39.551 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../05-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-22T20:19:39.551 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T20:19:39.570 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-22T20:19:39.578 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../06-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-22T20:19:39.579 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T20:19:39.595 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package librados2. 2026-02-22T20:19:39.603 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../07-librados2_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:39.603 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking librados2 (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:39.643 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libnbd0. 2026-02-22T20:19:39.651 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../08-libnbd0_1.10.5-1_amd64.deb ... 2026-02-22T20:19:39.652 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-22T20:19:39.666 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package librbd1. 2026-02-22T20:19:39.674 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../09-librbd1_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:39.674 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking librbd1 (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:39.720 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libcephfs2. 2026-02-22T20:19:39.728 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../10-libcephfs2_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:39.729 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libcephfs2 (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:39.751 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-rados. 2026-02-22T20:19:39.758 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../11-python3-rados_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:39.759 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-rados (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:39.776 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-22T20:19:39.784 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../12-python3-ceph-argparse_20.2.0-794-ga5c3084f-1jammy_all.deb ... 2026-02-22T20:19:39.784 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-ceph-argparse (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:39.798 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-cephfs. 2026-02-22T20:19:39.806 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../13-python3-cephfs_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:39.807 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-cephfs (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:39.822 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-22T20:19:39.830 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../14-python3-ceph-common_20.2.0-794-ga5c3084f-1jammy_all.deb ... 2026-02-22T20:19:39.830 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-ceph-common (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:39.848 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-22T20:19:39.855 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../15-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-22T20:19:39.856 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T20:19:39.871 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-prettytable. 2026-02-22T20:19:39.879 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../16-python3-prettytable_2.5.0-2_all.deb ... 2026-02-22T20:19:39.880 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-22T20:19:39.894 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-rbd. 2026-02-22T20:19:39.903 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../17-python3-rbd_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:39.903 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-rbd (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:39.920 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-22T20:19:39.928 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../18-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-22T20:19:39.928 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T20:19:39.946 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package librgw2. 2026-02-22T20:19:39.954 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../19-librgw2_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:39.955 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking librgw2 (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:40.047 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-rgw. 2026-02-22T20:19:40.055 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../20-python3-rgw_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:40.056 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-rgw (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:40.070 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-22T20:19:40.078 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../21-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-22T20:19:40.078 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T20:19:40.093 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libradosstriper1. 2026-02-22T20:19:40.101 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../22-libradosstriper1_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:40.102 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libradosstriper1 (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:40.118 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-common. 2026-02-22T20:19:40.126 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../23-ceph-common_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:40.126 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-common (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:40.388 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-base. 2026-02-22T20:19:40.396 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../24-ceph-base_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:40.401 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-base (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:40.465 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-22T20:19:40.474 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../25-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-22T20:19:40.474 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-22T20:19:40.487 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-cheroot. 2026-02-22T20:19:40.495 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../26-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-22T20:19:40.496 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T20:19:40.512 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-22T20:19:40.521 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../27-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-22T20:19:40.521 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-22T20:19:40.535 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-22T20:19:40.542 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../28-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-22T20:19:40.543 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-22T20:19:40.557 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-22T20:19:40.566 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../29-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-22T20:19:40.566 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-22T20:19:40.580 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-tempora. 2026-02-22T20:19:40.587 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../30-python3-tempora_4.1.2-1_all.deb ... 2026-02-22T20:19:40.588 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-22T20:19:40.602 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-portend. 2026-02-22T20:19:40.610 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../31-python3-portend_3.0.0-1_all.deb ... 2026-02-22T20:19:40.610 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-22T20:19:40.624 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-22T20:19:40.632 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../32-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-22T20:19:40.633 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-22T20:19:40.648 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-22T20:19:40.656 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../33-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-22T20:19:40.656 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-22T20:19:40.679 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-natsort. 2026-02-22T20:19:40.686 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../34-python3-natsort_8.0.2-1_all.deb ... 2026-02-22T20:19:40.687 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-22T20:19:40.702 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-22T20:19:40.710 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../35-ceph-mgr-modules-core_20.2.0-794-ga5c3084f-1jammy_all.deb ... 2026-02-22T20:19:40.710 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-mgr-modules-core (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:40.738 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-22T20:19:40.746 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../36-libsqlite3-mod-ceph_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:40.747 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libsqlite3-mod-ceph (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:40.762 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-mgr. 2026-02-22T20:19:40.770 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../37-ceph-mgr_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:40.770 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-mgr (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:40.791 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-mon. 2026-02-22T20:19:40.799 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../38-ceph-mon_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:40.799 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-mon (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:40.856 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-22T20:19:40.864 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../39-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-22T20:19:40.865 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T20:19:40.880 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-osd. 2026-02-22T20:19:40.888 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../40-ceph-osd_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:40.888 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-osd (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:41.031 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph. 2026-02-22T20:19:41.040 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../41-ceph_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:41.040 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:41.054 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-fuse. 2026-02-22T20:19:41.062 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../42-ceph-fuse_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:41.062 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-fuse (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:41.083 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-mds. 2026-02-22T20:19:41.092 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../43-ceph-mds_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:41.092 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-mds (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:41.124 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package cephadm. 2026-02-22T20:19:41.132 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../44-cephadm_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:41.132 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking cephadm (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:41.148 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-22T20:19:41.156 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../45-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T20:19:41.157 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T20:19:41.178 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-22T20:19:41.186 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../46-ceph-mgr-cephadm_20.2.0-794-ga5c3084f-1jammy_all.deb ... 2026-02-22T20:19:41.186 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-mgr-cephadm (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:41.206 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-22T20:19:41.214 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../47-python3-repoze.lru_0.7-2_all.deb ... 2026-02-22T20:19:41.215 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-22T20:19:41.230 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-routes. 2026-02-22T20:19:41.238 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../48-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-22T20:19:41.238 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T20:19:41.258 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-22T20:19:41.266 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../49-ceph-mgr-dashboard_20.2.0-794-ga5c3084f-1jammy_all.deb ... 2026-02-22T20:19:41.266 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-mgr-dashboard (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:41.694 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-22T20:19:41.703 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../50-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-22T20:19:41.703 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T20:19:41.741 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-joblib. 2026-02-22T20:19:41.749 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../51-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-22T20:19:41.750 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T20:19:41.776 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-22T20:19:41.785 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../52-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-22T20:19:41.785 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-22T20:19:41.799 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-sklearn. 2026-02-22T20:19:41.807 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../53-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-22T20:19:41.808 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T20:19:41.890 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-22T20:19:41.898 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../54-ceph-mgr-diskprediction-local_20.2.0-794-ga5c3084f-1jammy_all.deb ... 2026-02-22T20:19:41.899 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-mgr-diskprediction-local (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:42.061 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-cachetools. 2026-02-22T20:19:42.069 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../55-python3-cachetools_5.0.0-1_all.deb ... 2026-02-22T20:19:42.069 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-22T20:19:42.084 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-rsa. 2026-02-22T20:19:42.093 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../56-python3-rsa_4.8-1_all.deb ... 2026-02-22T20:19:42.093 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-22T20:19:42.111 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-google-auth. 2026-02-22T20:19:42.119 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../57-python3-google-auth_1.5.1-3_all.deb ... 2026-02-22T20:19:42.120 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-22T20:19:42.137 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-22T20:19:42.146 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../58-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-22T20:19:42.146 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T20:19:42.161 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-websocket. 2026-02-22T20:19:42.170 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../59-python3-websocket_1.2.3-1_all.deb ... 2026-02-22T20:19:42.170 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-22T20:19:42.188 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-22T20:19:42.196 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../60-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-22T20:19:42.197 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T20:19:42.296 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-22T20:19:42.305 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../61-ceph-mgr-k8sevents_20.2.0-794-ga5c3084f-1jammy_all.deb ... 2026-02-22T20:19:42.306 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-mgr-k8sevents (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:42.320 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-22T20:19:42.328 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../62-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-22T20:19:42.329 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T20:19:42.345 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-22T20:19:42.353 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../63-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T20:19:42.354 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T20:19:42.369 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package jq. 2026-02-22T20:19:42.377 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../64-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-22T20:19:42.377 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-22T20:19:42.392 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package socat. 2026-02-22T20:19:42.400 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../65-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-22T20:19:42.401 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-22T20:19:42.420 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package xmlstarlet. 2026-02-22T20:19:42.429 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../66-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-22T20:19:42.429 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-22T20:19:42.465 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-test. 2026-02-22T20:19:42.473 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../67-ceph-test_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:42.474 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-test (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:43.352 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package ceph-volume. 2026-02-22T20:19:43.361 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../68-ceph-volume_20.2.0-794-ga5c3084f-1jammy_all.deb ... 2026-02-22T20:19:43.361 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking ceph-volume (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:43.383 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libcephfs-daemon. 2026-02-22T20:19:43.391 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../69-libcephfs-daemon_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:43.392 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libcephfs-daemon (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:43.405 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libcephfs-proxy2. 2026-02-22T20:19:43.414 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../70-libcephfs-proxy2_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:43.414 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libcephfs-proxy2 (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:43.429 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-22T20:19:43.437 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../71-libcephfs-dev_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:43.437 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking libcephfs-dev (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:43.453 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package nvme-cli. 2026-02-22T20:19:43.462 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../72-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-22T20:19:43.462 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T20:19:43.494 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-22T20:19:43.503 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../73-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-22T20:19:43.503 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T20:19:43.531 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-22T20:19:43.540 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../74-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-22T20:19:43.540 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-22T20:19:43.555 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-pluggy. 2026-02-22T20:19:43.563 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../75-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-22T20:19:43.563 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-22T20:19:43.579 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-psutil. 2026-02-22T20:19:43.588 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../76-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-22T20:19:43.588 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-22T20:19:43.606 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-py. 2026-02-22T20:19:43.615 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../77-python3-py_1.10.0-1_all.deb ... 2026-02-22T20:19:43.615 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-22T20:19:43.635 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-pygments. 2026-02-22T20:19:43.644 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../78-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-22T20:19:43.644 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T20:19:43.686 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-toml. 2026-02-22T20:19:43.694 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../79-python3-toml_0.10.2-1_all.deb ... 2026-02-22T20:19:43.695 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-22T20:19:43.710 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-pytest. 2026-02-22T20:19:43.718 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../80-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-22T20:19:43.719 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T20:19:43.740 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-simplejson. 2026-02-22T20:19:43.748 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../81-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-22T20:19:43.749 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-22T20:19:43.767 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package python3-webob. 2026-02-22T20:19:43.775 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../82-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-22T20:19:43.775 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T20:19:43.798 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-22T20:19:43.806 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../83-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-22T20:19:43.807 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-22T20:19:43.875 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package radosgw. 2026-02-22T20:19:43.884 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../84-radosgw_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:43.884 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking radosgw (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:44.089 INFO:teuthology.orchestra.run.trial150.stdout:Selecting previously unselected package rbd-fuse. 2026-02-22T20:19:44.098 INFO:teuthology.orchestra.run.trial150.stdout:Preparing to unpack .../85-rbd-fuse_20.2.0-794-ga5c3084f-1jammy_amd64.deb ... 2026-02-22T20:19:44.098 INFO:teuthology.orchestra.run.trial150.stdout:Unpacking rbd-fuse (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:44.125 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-22T20:19:44.193 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-22T20:19:44.193 INFO:teuthology.orchestra.run.trial150.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-22T20:19:44.265 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-22T20:19:44.538 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-22T20:19:44.905 INFO:teuthology.orchestra.run.trial150.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-22T20:19:44.916 INFO:teuthology.orchestra.run.trial150.stdout:Setting up cephadm (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:44.957 INFO:teuthology.orchestra.run.trial150.stdout:Adding system user cephadm....done 2026-02-22T20:19:44.968 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-22T20:19:45.032 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-22T20:19:45.033 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-22T20:19:45.105 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-22T20:19:45.181 INFO:teuthology.orchestra.run.trial150.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-22T20:19:45.182 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-22T20:19:45.276 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-22T20:19:45.399 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-22T20:19:45.472 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-22T20:19:45.546 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-ceph-argparse (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:45.611 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-22T20:19:45.612 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-22T20:19:45.613 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-22T20:19:45.613 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-22T20:19:45.614 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-22T20:19:45.735 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-22T20:19:45.811 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libcephfs-proxy2 (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:45.811 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-22T20:19:45.890 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-22T20:19:45.981 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-22T20:19:46.228 INFO:teuthology.orchestra.run.trial150.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-22T20:19:46.229 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-22T20:19:46.324 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-22T20:19:46.449 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-22T20:19:46.529 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-22T20:19:46.602 INFO:teuthology.orchestra.run.trial150.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-22T20:19:46.603 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-ceph-common (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:46.690 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-22T20:19:47.180 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T20:19:47.184 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-22T20:19:47.262 INFO:teuthology.orchestra.run.trial150.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-22T20:19:47.262 INFO:teuthology.orchestra.run.trial150.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-22T20:19:47.263 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-22T20:19:47.340 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-22T20:19:47.413 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T20:19:47.414 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-22T20:19:47.490 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-22T20:19:47.563 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-22T20:19:47.639 INFO:teuthology.orchestra.run.trial150.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-22T20:19:47.640 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-22T20:19:47.720 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-22T20:19:47.721 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-22T20:19:47.797 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-22T20:19:47.882 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-22T20:19:47.954 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-22T20:19:47.955 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-22T20:19:48.090 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-22T20:19:48.161 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-22T20:19:48.162 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-22T20:19:48.243 INFO:teuthology.orchestra.run.trial150.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-22T20:19:48.244 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-22T20:19:48.375 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-22T20:19:48.376 INFO:teuthology.orchestra.run.trial150.stdout:Setting up librados2 (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:48.377 INFO:teuthology.orchestra.run.trial150.stdout:Setting up librgw2 (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:48.378 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libsqlite3-mod-ceph (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:48.379 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-22T20:19:48.892 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libcephfs2 (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:48.893 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libradosstriper1 (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:48.894 INFO:teuthology.orchestra.run.trial150.stdout:Setting up librbd1 (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:48.894 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-mgr-modules-core (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:48.895 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-fuse (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:48.971 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T20:19:48.971 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-22T20:19:49.350 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libcephfs-dev (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:49.351 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-rados (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:49.352 INFO:teuthology.orchestra.run.trial150.stdout:Setting up libcephfs-daemon (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:49.353 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-rbd (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:49.353 INFO:teuthology.orchestra.run.trial150.stdout:Setting up rbd-fuse (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:49.354 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-rgw (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:49.355 INFO:teuthology.orchestra.run.trial150.stdout:Setting up python3-cephfs (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:49.356 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-common (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:49.385 INFO:teuthology.orchestra.run.trial150.stdout:Adding group ceph....done 2026-02-22T20:19:49.419 INFO:teuthology.orchestra.run.trial150.stdout:Adding system user ceph....done 2026-02-22T20:19:49.429 INFO:teuthology.orchestra.run.trial150.stdout:Setting system user ceph properties....done 2026-02-22T20:19:49.435 INFO:teuthology.orchestra.run.trial150.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-22T20:19:49.522 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-22T20:19:49.789 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-22T20:19:50.147 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-test (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:50.148 INFO:teuthology.orchestra.run.trial150.stdout:Setting up radosgw (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:50.400 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T20:19:50.400 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-22T20:19:50.766 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-base (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:50.883 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-22T20:19:51.244 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-mds (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:51.304 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T20:19:51.304 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-22T20:19:51.647 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-mgr (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:51.721 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T20:19:51.721 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-22T20:19:52.047 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-osd (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:52.129 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T20:19:52.129 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-22T20:19:52.491 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-mgr-k8sevents (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:52.501 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-mgr-diskprediction-local (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:52.514 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-mon (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:52.572 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T20:19:52.572 INFO:teuthology.orchestra.run.trial150.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-22T20:19:52.954 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-mgr-cephadm (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:52.964 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:52.964 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-mgr-dashboard (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:52.974 INFO:teuthology.orchestra.run.trial150.stdout:Setting up ceph-volume (20.2.0-794-ga5c3084f-1jammy) ... 2026-02-22T20:19:53.080 INFO:teuthology.orchestra.run.trial150.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-22T20:19:53.100 INFO:teuthology.orchestra.run.trial150.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-22T20:19:53.755 INFO:teuthology.orchestra.run.trial150.stdout: 2026-02-22T20:19:53.756 INFO:teuthology.orchestra.run.trial150.stdout:Running kernel seems to be up-to-date. 2026-02-22T20:19:53.756 INFO:teuthology.orchestra.run.trial150.stdout: 2026-02-22T20:19:53.756 INFO:teuthology.orchestra.run.trial150.stdout:The processor microcode seems to be up-to-date. 2026-02-22T20:19:53.756 INFO:teuthology.orchestra.run.trial150.stdout: 2026-02-22T20:19:53.756 INFO:teuthology.orchestra.run.trial150.stdout:No services need to be restarted. 2026-02-22T20:19:53.756 INFO:teuthology.orchestra.run.trial150.stdout: 2026-02-22T20:19:53.756 INFO:teuthology.orchestra.run.trial150.stdout:No containers need to be restarted. 2026-02-22T20:19:53.756 INFO:teuthology.orchestra.run.trial150.stdout: 2026-02-22T20:19:53.757 INFO:teuthology.orchestra.run.trial150.stdout:No user sessions are running outdated binaries. 2026-02-22T20:19:53.757 INFO:teuthology.orchestra.run.trial150.stdout: 2026-02-22T20:19:53.757 INFO:teuthology.orchestra.run.trial150.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-22T20:19:54.545 INFO:teuthology.orchestra.run.trial150.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-22T20:19:54.551 DEBUG:teuthology.parallel:result is None 2026-02-22T20:19:54.552 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=a5c3084f688b1f6eff223b8d259a8316f9844c8e 2026-02-22T20:19:54.722 DEBUG:teuthology.orchestra.run.trial150:> dpkg-query -W -f '${Version}' ceph 2026-02-22T20:19:54.733 INFO:teuthology.orchestra.run.trial150.stdout:20.2.0-794-ga5c3084f-1jammy 2026-02-22T20:19:54.733 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-794-ga5c3084f-1jammy 2026-02-22T20:19:54.733 INFO:teuthology.task.install:The correct ceph version 20.2.0-794-ga5c3084f-1jammy is installed. 2026-02-22T20:19:54.735 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-22T20:19:54.735 DEBUG:teuthology.orchestra.run.trial150:> set -ex 2026-02-22T20:19:54.735 DEBUG:teuthology.orchestra.run.trial150:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-22T20:19:54.785 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-22T20:19:54.787 DEBUG:teuthology.orchestra.run.trial150:> set -ex 2026-02-22T20:19:54.787 DEBUG:teuthology.orchestra.run.trial150:> sudo dd of=/usr/bin/daemon-helper 2026-02-22T20:19:54.836 DEBUG:teuthology.orchestra.run.trial150:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-22T20:19:54.883 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-22T20:19:54.884 DEBUG:teuthology.orchestra.run.trial150:> set -ex 2026-02-22T20:19:54.884 DEBUG:teuthology.orchestra.run.trial150:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-22T20:19:54.931 DEBUG:teuthology.orchestra.run.trial150:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-22T20:19:54.980 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-22T20:19:54.980 DEBUG:teuthology.orchestra.run.trial150:> set -ex 2026-02-22T20:19:54.981 DEBUG:teuthology.orchestra.run.trial150:> sudo dd of=/usr/bin/stdin-killer 2026-02-22T20:19:55.028 DEBUG:teuthology.orchestra.run.trial150:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-22T20:19:55.076 INFO:teuthology.run_tasks:Running task workunit... 2026-02-22T20:19:55.108 INFO:tasks.workunit:Pulling workunits from ref a5c3084f688b1f6eff223b8d259a8316f9844c8e 2026-02-22T20:19:55.108 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-22T20:19:55.109 INFO:tasks.workunit:timeout=3h 2026-02-22T20:19:55.109 INFO:tasks.workunit:cleanup=True 2026-02-22T20:19:55.109 DEBUG:teuthology.orchestra.run.trial150:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-22T20:19:55.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T20:19:55.125 INFO:teuthology.orchestra.run.trial150.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-22T20:19:55.125 DEBUG:teuthology.orchestra.run.trial150:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-22T20:19:55.172 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-22T20:19:55.173 DEBUG:teuthology.orchestra.run.trial150:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-22T20:19:55.217 DEBUG:teuthology.orchestra.run.trial150:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a5c3084f688b1f6eff223b8d259a8316f9844c8e 2026-02-22T20:19:55.267 INFO:tasks.workunit.client.0.trial150.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-22T20:20:24.377 INFO:tasks.workunit.client.0.trial150.stderr:Note: switching to 'a5c3084f688b1f6eff223b8d259a8316f9844c8e'. 2026-02-22T20:20:24.378 INFO:tasks.workunit.client.0.trial150.stderr: 2026-02-22T20:20:24.378 INFO:tasks.workunit.client.0.trial150.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-22T20:20:24.378 INFO:tasks.workunit.client.0.trial150.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-22T20:20:24.378 INFO:tasks.workunit.client.0.trial150.stderr:state without impacting any branches by switching back to a branch. 2026-02-22T20:20:24.378 INFO:tasks.workunit.client.0.trial150.stderr: 2026-02-22T20:20:24.378 INFO:tasks.workunit.client.0.trial150.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-22T20:20:24.378 INFO:tasks.workunit.client.0.trial150.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-22T20:20:24.378 INFO:tasks.workunit.client.0.trial150.stderr: 2026-02-22T20:20:24.378 INFO:tasks.workunit.client.0.trial150.stderr: git switch -c 2026-02-22T20:20:24.379 INFO:tasks.workunit.client.0.trial150.stderr: 2026-02-22T20:20:24.379 INFO:tasks.workunit.client.0.trial150.stderr:Or undo this operation with: 2026-02-22T20:20:24.379 INFO:tasks.workunit.client.0.trial150.stderr: 2026-02-22T20:20:24.379 INFO:tasks.workunit.client.0.trial150.stderr: git switch - 2026-02-22T20:20:24.379 INFO:tasks.workunit.client.0.trial150.stderr: 2026-02-22T20:20:24.379 INFO:tasks.workunit.client.0.trial150.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-22T20:20:24.379 INFO:tasks.workunit.client.0.trial150.stderr: 2026-02-22T20:20:24.379 INFO:tasks.workunit.client.0.trial150.stderr:HEAD is now at a5c3084f688 Integration branch metadata 2026-02-22T20:20:24.389 DEBUG:teuthology.orchestra.run.trial150:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-22T20:20:24.435 DEBUG:teuthology.orchestra.run.trial150:> set -ex 2026-02-22T20:20:24.435 DEBUG:teuthology.orchestra.run.trial150:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-22T20:20:24.482 INFO:tasks.workunit:Running workunits matching scrub on client.0... 2026-02-22T20:20:24.482 INFO:tasks.workunit:Running workunit scrub/osd-mapper.sh... 2026-02-22T20:20:24.483 DEBUG:teuthology.orchestra.run.trial150:workunit test scrub/osd-mapper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5c3084f688b1f6eff223b8d259a8316f9844c8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh 2026-02-22T20:20:24.535 INFO:tasks.workunit.client.0.trial150.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T20:20:24.541 INFO:tasks.workunit.client.0.trial150.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T20:20:24.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T20:20:24.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T20:20:24.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-22T20:20:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-22T20:20:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-22T20:20:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-22T20:20:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-22T20:20:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-22T20:20:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-mapper 2026-02-22T20:20:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:11: run: local dir=td/osd-mapper 2026-02-22T20:20:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:12: run: shift 2026-02-22T20:20:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: export CEPH_MON=127.0.0.1:7144 2026-02-22T20:20:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:14: run: CEPH_MON=127.0.0.1:7144 2026-02-22T20:20:24.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:15: run: export CEPH_ARGS 2026-02-22T20:20:24.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: uuidgen 2026-02-22T20:20:24.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:16: run: CEPH_ARGS+='--fsid=77a8cc62-b008-4f99-be84-6b5f87b5e02b --auth-supported=none ' 2026-02-22T20:20:24.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:17: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2026-02-22T20:20:24.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:19: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-22T20:20:24.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: set 2026-02-22T20:20:24.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T20:20:24.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:20: run: local funcs=TEST_truncated_sna_record 2026-02-22T20:20:24.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:21: run: for func in $funcs 2026-02-22T20:20:24.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:22: run: setup td/osd-mapper 2026-02-22T20:20:24.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-mapper 2026-02-22T20:20:24.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-mapper 2026-02-22T20:20:24.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2026-02-22T20:20:24.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:20:24.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2026-02-22T20:20:24.546 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:20:24.546 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:20:24.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:20:24.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:20:24.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:20:24.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:20:24.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:20:24.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:20:24.548 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:20:24.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:20:24.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:20:24.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:20:24.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:20:24.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:20:24.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:20:24.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:20:24.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:20:24.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:20:24.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2026-02-22T20:20:24.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:20:24.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:20:24.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12039 2026-02-22T20:20:24.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12039 2026-02-22T20:20:24.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:20:24.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:20:24.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-mapper 2026-02-22T20:20:24.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:20:24.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:20:24.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12039 2026-02-22T20:20:24.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.12039 2026-02-22T20:20:24.560 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:20:24.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-22T20:20:24.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-22T20:20:24.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:20:24.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-mapper 1' TERM HUP INT 2026-02-22T20:20:24.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:23: run: TEST_truncated_sna_record td/osd-mapper 2026-02-22T20:20:24.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:45: TEST_truncated_sna_record: local dir=td/osd-mapper 2026-02-22T20:20:24.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2026-02-22T20:20:24.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:46: TEST_truncated_sna_record: local -A cluster_conf 2026-02-22T20:20:24.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:52: TEST_truncated_sna_record: local extr_dbg=3 2026-02-22T20:20:24.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: (( extr_dbg > 1 )) 2026-02-22T20:20:24.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:53: TEST_truncated_sna_record: echo 'Dir: td/osd-mapper' 2026-02-22T20:20:24.563 INFO:tasks.workunit.client.0.trial150.stdout:Dir: td/osd-mapper 2026-02-22T20:20:24.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:54: TEST_truncated_sna_record: standard_scrub_cluster td/osd-mapper cluster_conf 2026-02-22T20:20:24.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-mapper 2026-02-22T20:20:24.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-22T20:20:24.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2026-02-22T20:20:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2026-02-22T20:20:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=test 2026-02-22T20:20:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=test 2026-02-22T20:20:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=3 2026-02-22T20:20:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=3 2026-02-22T20:20:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local extra_pars= 2026-02-22T20:20:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-22T20:20:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-22T20:20:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-22T20:20:24.850 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 179089d4-c743-481a-962e-b2f009b9b384 2026-02-22T20:20:24.952 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:20:24.980 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:24.975+0000 7fe8f985da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:24.981 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:24.979+0000 7fe8f985da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:24.982 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:24.979+0000 7fe8f985da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:24.983 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:24.979+0000 7fe8f985da40 -1 bdev(0x55f70018ec00 td/osd-mapper/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:20:24.983 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:24.979+0000 7fe8f985da40 -1 bluestore(td/osd-mapper/0) _read_fsid unparsable uuid 2026-02-22T20:20:25.688 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:20:25.779 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:20:25.798 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:25.795+0000 7f1ba862da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:25.800 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:25.795+0000 7f1ba862da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:25.801 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:25.795+0000 7f1ba862da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:25.934 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:20:26.183 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:26.179+0000 7f1ba862da40 -1 Falling back to public interface 2026-02-22T20:20:26.738 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:26.735+0000 7f1ba862da40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:20:27.095 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:20:28.261 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:20:28.422 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2318802836,v1:127.0.0.1:6803/2318802836] [v2:127.0.0.1:6804/2318802836,v1:127.0.0.1:6805/2318802836] exists,up 179089d4-c743-481a-962e-b2f009b9b384 2026-02-22T20:20:28.427 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 228fdfd9-fd50-46e6-8a9f-1b129b6336bc 2026-02-22T20:20:28.597 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:20:28.626 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:28.623+0000 7f026eb0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:28.627 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:28.623+0000 7f026eb0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:28.628 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:28.623+0000 7f026eb0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:28.629 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:28.623+0000 7f026eb0ea40 -1 bdev(0x55f1268eec00 td/osd-mapper/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:20:28.629 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:28.623+0000 7f026eb0ea40 -1 bluestore(td/osd-mapper/1) _read_fsid unparsable uuid 2026-02-22T20:20:29.254 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:20:29.457 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:20:29.476 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:29.471+0000 7f232903ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:29.477 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:29.475+0000 7f232903ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:29.478 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:29.475+0000 7f232903ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:29.613 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:20:29.835 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:29.831+0000 7f232903ea40 -1 Falling back to public interface 2026-02-22T20:20:30.441 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:30.435+0000 7f232903ea40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:20:30.771 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:20:31.926 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:20:32.077 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/414277883,v1:127.0.0.1:6811/414277883] [v2:127.0.0.1:6812/414277883,v1:127.0.0.1:6813/414277883] exists,up 228fdfd9-fd50-46e6-8a9f-1b129b6336bc 2026-02-22T20:20:32.083 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 b8b14a6f-e85a-4d06-adb1-6da13b14be92 2026-02-22T20:20:32.259 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:20:32.288 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:32.283+0000 7fd7d9f6ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:32.289 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:32.287+0000 7fd7d9f6ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:32.290 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:32.287+0000 7fd7d9f6ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:32.291 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:32.287+0000 7fd7d9f6ca40 -1 bdev(0x559f808fcc00 td/osd-mapper/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:20:32.291 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:32.287+0000 7fd7d9f6ca40 -1 bluestore(td/osd-mapper/2) _read_fsid unparsable uuid 2026-02-22T20:20:32.949 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:20:33.155 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:20:33.184 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:33.179+0000 7ffb7958ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:33.186 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:33.183+0000 7ffb7958ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:33.187 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:33.183+0000 7ffb7958ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:33.310 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:20:33.563 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:33.559+0000 7ffb7958ca40 -1 Falling back to public interface 2026-02-22T20:20:34.203 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:34.198+0000 7ffb7958ca40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:20:34.466 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:20:35.630 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:20:35.787 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/858143858,v1:127.0.0.1:6819/858143858] [v2:127.0.0.1:6820/858143858,v1:127.0.0.1:6821/858143858] exists,up b8b14a6f-e85a-4d06-adb1-6da13b14be92 2026-02-22T20:20:35.996 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T20:20:37.483 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836491 2026-02-22T20:20:38.817 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705671 2026-02-22T20:20:38.988 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574852 2026-02-22T20:20:40.855 INFO:tasks.workunit.client.0.trial150.stdout:standard_scrub_cluster: dbg: test pool is test 1 2026-02-22T20:20:40.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:55: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2026-02-22T20:20:40.922 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:20:40.923 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-22T20:20:40.923 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:20:40.930 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:20:40.930 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-22T20:20:40.930 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:20:40.938 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:20:40.938 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-22T20:20:40.938 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:20:40.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:56: TEST_truncated_sna_record: ceph tell 'osd.*' config set osd_stats_update_period_scrubbing 1 2026-02-22T20:20:41.022 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:20:41.022 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_stats_update_period_scrubbing = '' " 2026-02-22T20:20:41.022 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:20:41.029 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:20:41.030 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_stats_update_period_scrubbing = '' " 2026-02-22T20:20:41.030 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:20:41.037 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:20:41.037 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_stats_update_period_scrubbing = '' " 2026-02-22T20:20:41.037 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:20:41.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:58: TEST_truncated_sna_record: local osdn=3 2026-02-22T20:20:41.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:59: TEST_truncated_sna_record: local poolid=1 2026-02-22T20:20:41.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:60: TEST_truncated_sna_record: local poolname=test 2026-02-22T20:20:41.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:61: TEST_truncated_sna_record: local objname=objxxx 2026-02-22T20:20:41.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:64: TEST_truncated_sna_record: make_a_clone test objxxx snap01 snap02 2026-02-22T20:20:41.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2026-02-22T20:20:41.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2026-02-22T20:20:41.098 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap01 2026-02-22T20:20:41.200 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap02 2026-02-22T20:20:41.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:65: TEST_truncated_sna_record: make_a_clone test objxxx snap13 2026-02-22T20:20:41.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2026-02-22T20:20:41.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2026-02-22T20:20:41.303 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap13 2026-02-22T20:20:41.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:66: TEST_truncated_sna_record: make_a_clone test objxxx snap24 snap25 2026-02-22T20:20:41.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:32: make_a_clone: local saved_echo_flag=x 2026-02-22T20:20:41.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:33: make_a_clone: set +x 2026-02-22T20:20:41.405 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap24 2026-02-22T20:20:41.506 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap25 2026-02-22T20:20:41.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: echo 12405 2026-02-22T20:20:41.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:67: TEST_truncated_sna_record: rados -p test put objxxx - 2026-02-22T20:20:41.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2026-02-22T20:20:41.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: jq -r .pgid 2026-02-22T20:20:41.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:70: TEST_truncated_sna_record: local pgid=1.3 2026-02-22T20:20:41.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2026-02-22T20:20:41.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: jq -r '.up[0]' 2026-02-22T20:20:41.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:71: TEST_truncated_sna_record: local osd=1 2026-02-22T20:20:41.842 INFO:tasks.workunit.client.0.trial150.stdout:pgid is 1.3 (primary: osd.1) 2026-02-22T20:20:41.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:72: TEST_truncated_sna_record: echo 'pgid is 1.3 (primary: osd.1)' 2026-02-22T20:20:41.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:74: TEST_truncated_sna_record: set_query_debug 1.3 2026-02-22T20:20:41.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:417: set_query_debug: local pgid=1.3 2026-02-22T20:20:41.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: ceph pg dump pgs_brief 2026-02-22T20:20:41.844 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: awk -v 'pg=^1.3' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2026-02-22T20:20:41.987 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs_brief 2026-02-22T20:20:41.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: local prim_osd=1 2026-02-22T20:20:41.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:421: set_query_debug: echo 'Setting scrub debug data. Primary for 1.3 is 1' 2026-02-22T20:20:41.996 INFO:tasks.workunit.client.0.trial150.stdout:Setting scrub debug data. Primary for 1.3 is 1 2026-02-22T20:20:41.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: get_asok_path osd.1 2026-02-22T20:20:41.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:20:41.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:20:41.997 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:20:41.997 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:20:41.997 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12039 2026-02-22T20:20:41.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.12039/ceph-osd.1.asok 2026-02-22T20:20:41.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: CEPH_ARGS= 2026-02-22T20:20:41.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.12039/ceph-osd.1.asok scrubdebug 1.3 set sessions 2026-02-22T20:20:42.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:20:42.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:77: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout:{"success":true}{ 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "name": "objxxx", 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 5, 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "id": 2, 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "snapshots": [ 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "id": 1, 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "name": "snap01" 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "id": 2, 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "name": "snap02" 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "size": 6, 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "overlaps": [] 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "id": 3, 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "snapshots": [ 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "id": 3, 2026-02-22T20:20:42.086 INFO:tasks.workunit.client.0.trial150.stdout: "name": "snap13" 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "size": 6, 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "overlaps": [] 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "id": 5, 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "snapshots": [ 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "id": 4, 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "name": "snap24" 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "id": 5, 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "name": "snap25" 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "size": 6, 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "overlaps": [] 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "id": "head", 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "snapshots": [], 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: "size": 6 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:20:42.087 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:20:42.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:80: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2026-02-22T20:20:42.153 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:20:42.153 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:20:42.154 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:20:42.154 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:20:42.154 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:20:42.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:84: TEST_truncated_sna_record: sleep 3 2026-02-22T20:20:45.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:85: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-22T20:20:45.310 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:20:45.310 INFO:tasks.workunit.client.0.trial150.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2026-02-22T20:20:42.326896+0000 22'7 22:43 [1,2,0] 1 [1,2,0] 1 22'7 2026-02-22T20:20:42.326834+0000 22'7 2026-02-22T20:20:42.326834+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:42.326834+0000 4 0 2026-02-22T20:20:45.311 INFO:tasks.workunit.client.0.trial150.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:20:36.098733+0000 0'0 22:30 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:20:45.311 INFO:tasks.workunit.client.0.trial150.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:20:36.505420+0000 0'0 22:27 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:20:45.311 INFO:tasks.workunit.client.0.trial150.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:20:36.098576+0000 0'0 22:32 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:20:45.311 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:45.311 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:20:45.311 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:20:45.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:86: TEST_truncated_sna_record: grep -a -q -- 'event: --^^^^---- ScrubFinished' td/osd-mapper/osd.1.log 2026-02-22T20:20:45.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:90: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-22T20:20:45.465 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:20:45.466 INFO:tasks.workunit.client.0.trial150.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2026-02-22T20:20:42.326896+0000 22'7 22:43 [1,2,0] 1 [1,2,0] 1 22'7 2026-02-22T20:20:42.326834+0000 22'7 2026-02-22T20:20:42.326834+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:42.326834+0000 4 0 2026-02-22T20:20:45.466 INFO:tasks.workunit.client.0.trial150.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:20:36.098733+0000 0'0 22:30 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:20:45.466 INFO:tasks.workunit.client.0.trial150.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:20:36.505420+0000 0'0 22:27 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:20:45.466 INFO:tasks.workunit.client.0.trial150.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:20:36.098576+0000 0'0 22:32 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:20:45.466 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:45.466 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:20:45.466 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:20:45.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:91: TEST_truncated_sna_record: ceph osd set noscrub 2026-02-22T20:20:45.679 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:20:45.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:92: TEST_truncated_sna_record: ceph osd set nodeep-scrub 2026-02-22T20:20:45.886 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:20:45.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:93: TEST_truncated_sna_record: sleep 5 2026-02-22T20:20:50.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:94: TEST_truncated_sna_record: grep -a -q -v ERR td/osd-mapper/osd.1.log 2026-02-22T20:20:50.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:97: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2026-02-22T20:20:50.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:20:50.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:20:50.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:20:50.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:20:50.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:20:51.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:20:51.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-22T20:20:51.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:99: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2026-02-22T20:20:51.224 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.218+0000 7f4cefef2d40 1 bdev(0x55d832f43800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-22T20:20:51.224 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.218+0000 7f4cefef2d40 1 bdev(0x55d832f43800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:51.224 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.218+0000 7f4cefef2d40 1 bdev(0x55d832f43800 td/osd-mapper/0/block) close 2026-02-22T20:20:51.278 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.274+0000 7f4cefef2d40 1 bdev(0x55d832f43800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-22T20:20:51.278 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.274+0000 7f4cefef2d40 1 bdev(0x55d832f43800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:51.279 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.274+0000 7f4cefef2d40 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T20:20:51.279 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.274+0000 7f4cefef2d40 1 bdev(0x55d832f43c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-22T20:20:51.279 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.274+0000 7f4cefef2d40 1 bdev(0x55d832f43c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:51.279 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.274+0000 7f4cefef2d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2026-02-22T20:20:51.279 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.274+0000 7f4cefef2d40 1 bluefs mount 2026-02-22T20:20:51.279 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.274+0000 7f4cefef2d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T20:20:51.282 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.278+0000 7f4cefef2d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.282 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.278+0000 7f4cefef2d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.282 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.278+0000 7f4cefef2d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.282 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.278+0000 7f4cefef2d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.282 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.278+0000 7f4cefef2d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.283 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.278+0000 7f4cefef2d40 1 bluefs mount shared_bdev_used = 0 2026-02-22T20:20:51.283 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.278+0000 7f4cefef2d40 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T20:20:51.295 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.290+0000 7f4cefef2d40 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T20:20:51.295 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.290+0000 7f4cefef2d40 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2026-02-22T20:20:51.295 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.290+0000 7f4cefef2d40 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2026-02-22T20:20:51.296 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.290+0000 7f4cefef2d40 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T20:20:51.296 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.290+0000 7f4cefef2d40 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2026-02-22T20:20:51.296 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.290+0000 7f4cefef2d40 1 freelist init 2026-02-22T20:20:51.296 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.290+0000 7f4cefef2d40 1 freelist _read_cfg 2026-02-22T20:20:51.296 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.290+0000 7f4cefef2d40 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-22T20:20:51.297 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.290+0000 7f4cefef2d40 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 3 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc8000, fragmentation 7.62941e-08 2026-02-22T20:20:51.297 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.290+0000 7f4cefef2d40 1 bluefs umount 2026-02-22T20:20:51.297 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.290+0000 7f4cefef2d40 1 bdev(0x55d832f43c00 td/osd-mapper/0/block) close 2026-02-22T20:20:51.350 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.346+0000 7f4cefef2d40 1 bdev(0x55d832f43c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-22T20:20:51.350 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.346+0000 7f4cefef2d40 1 bdev(0x55d832f43c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:51.350 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.346+0000 7f4cefef2d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2026-02-22T20:20:51.350 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.346+0000 7f4cefef2d40 1 bluefs mount 2026-02-22T20:20:51.351 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.346+0000 7f4cefef2d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T20:20:51.352 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.346+0000 7f4cefef2d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.352 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.346+0000 7f4cefef2d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.352 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.346+0000 7f4cefef2d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.352 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.346+0000 7f4cefef2d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.352 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.346+0000 7f4cefef2d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.352 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.346+0000 7f4cefef2d40 1 bluefs mount shared_bdev_used = 27656192 2026-02-22T20:20:51.352 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.346+0000 7f4cefef2d40 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T20:20:51.364 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.358+0000 7f4cefef2d40 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T20:20:51.364 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-22T20:20:51.365 INFO:tasks.workunit.client.0.trial150.stdout:00000000 0b 05 58 02 00 00 77 a8 cc 62 b0 08 4f 99 be 84 |..X...w..b..O...| 2026-02-22T20:20:51.365 INFO:tasks.workunit.client.0.trial150.stdout:00000010 6b 5f 87 b5 e0 2b 00 00 00 00 18 00 00 00 00 00 |k_...+..........| 2026-02-22T20:20:51.365 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-22T20:20:51.365 INFO:tasks.workunit.client.0.trial150.stdout:* 2026-02-22T20:20:51.365 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-22T20:20:51.365 INFO:tasks.workunit.client.0.trial150.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-22T20:20:51.365 INFO:tasks.workunit.client.0.trial150.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-22T20:20:51.365 INFO:tasks.workunit.client.0.trial150.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-22T20:20:51.365 INFO:tasks.workunit.client.0.trial150.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-22T20:20:51.365 INFO:tasks.workunit.client.0.trial150.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-22T20:20:51.366 INFO:tasks.workunit.client.0.trial150.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-22T20:20:51.366 INFO:tasks.workunit.client.0.trial150.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-22T20:20:51.366 INFO:tasks.workunit.client.0.trial150.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-22T20:20:51.366 INFO:tasks.workunit.client.0.trial150.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-22T20:20:51.366 INFO:tasks.workunit.client.0.trial150.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-22T20:20:51.366 INFO:tasks.workunit.client.0.trial150.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-22T20:20:51.366 INFO:tasks.workunit.client.0.trial150.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-22T20:20:51.366 INFO:tasks.workunit.client.0.trial150.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-22T20:20:51.366 INFO:tasks.workunit.client.0.trial150.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-22T20:20:51.366 INFO:tasks.workunit.client.0.trial150.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-22T20:20:51.366 INFO:tasks.workunit.client.0.trial150.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-22T20:20:51.366 INFO:tasks.workunit.client.0.trial150.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-22T20:20:51.367 INFO:tasks.workunit.client.0.trial150.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-22T20:20:51.367 INFO:tasks.workunit.client.0.trial150.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-22T20:20:51.367 INFO:tasks.workunit.client.0.trial150.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-22T20:20:51.367 INFO:tasks.workunit.client.0.trial150.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-22T20:20:51.367 INFO:tasks.workunit.client.0.trial150.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-22T20:20:51.367 INFO:tasks.workunit.client.0.trial150.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-22T20:20:51.367 INFO:tasks.workunit.client.0.trial150.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-22T20:20:51.367 INFO:tasks.workunit.client.0.trial150.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-22T20:20:51.367 INFO:tasks.workunit.client.0.trial150.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-22T20:20:51.367 INFO:tasks.workunit.client.0.trial150.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-22T20:20:51.367 INFO:tasks.workunit.client.0.trial150.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-22T20:20:51.367 INFO:tasks.workunit.client.0.trial150.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-22T20:20:51.368 INFO:tasks.workunit.client.0.trial150.stdout:00000220 72 65 18 00 00 00 05 00 00 00 17 90 89 d4 c7 43 |re.............C| 2026-02-22T20:20:51.368 INFO:tasks.workunit.client.0.trial150.stdout:00000230 48 1a 96 2e b2 f0 09 b9 b3 84 00 00 00 00 00 00 |H...............| 2026-02-22T20:20:51.368 INFO:tasks.workunit.client.0.trial150.stdout:00000240 00 00 18 00 00 00 0b 65 9b 69 8e 81 98 2e 01 00 |.......e.i......| 2026-02-22T20:20:51.368 INFO:tasks.workunit.client.0.trial150.stdout:00000250 00 00 01 00 00 00 01 00 00 00 18 00 00 00 |..............| 2026-02-22T20:20:51.368 INFO:tasks.workunit.client.0.trial150.stdout:0000025e 2026-02-22T20:20:51.368 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.368 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-22T20:20:51.368 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.368 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:51.368 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-22T20:20:51.368 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:51.368 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:51.369 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-22T20:20:51.369 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-22T20:20:51.369 INFO:tasks.workunit.client.0.trial150.stdout:00000047 2026-02-22T20:20:51.369 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.369 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:51.369 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-22T20:20:51.369 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:51.369 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:51.369 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-22T20:20:51.369 INFO:tasks.workunit.client.0.trial150.stdout:0000003f 2026-02-22T20:20:51.369 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.370 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:51.370 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-22T20:20:51.370 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:51.370 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:51.370 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-22T20:20:51.370 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-22T20:20:51.370 INFO:tasks.workunit.client.0.trial150.stdout:00000047 2026-02-22T20:20:51.370 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.370 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:51.370 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:51.370 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:51.370 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:51.371 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:51.371 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:51.371 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.371 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:51.371 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:51.371 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:51.371 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:51.371 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:51.371 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:51.371 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.371 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:51.371 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:51.372 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:51.372 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:51.372 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:51.372 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:51.372 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.372 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:51.372 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:51.372 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:51.372 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:51.372 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:51.372 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:51.372 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.373 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:51.373 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:51.373 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:51.373 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:51.373 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:51.373 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:51.373 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.373 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-22T20:20:51.373 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.373 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-22T20:20:51.373 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.374 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.358+0000 7f4cefef2d40 1 bluefs umount 2026-02-22T20:20:51.374 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.358+0000 7f4cefef2d40 1 bdev(0x55d832f43c00 td/osd-mapper/0/block) close 2026-02-22T20:20:51.418 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.414+0000 7f4cefef2d40 1 freelist shutdown 2026-02-22T20:20:51.418 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.414+0000 7f4cefef2d40 1 bdev(0x55d832f43800 td/osd-mapper/0/block) close 2026-02-22T20:20:51.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-22T20:20:51.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2026-02-22T20:20:51.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:100: TEST_truncated_sna_record: grep -a SNA_ 2026-02-22T20:20:51.474 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.470+0000 7fa75a5b3d40 1 bdev(0x55f2c2231800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-22T20:20:51.475 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.470+0000 7fa75a5b3d40 1 bdev(0x55f2c2231800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:51.475 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.470+0000 7fa75a5b3d40 1 bdev(0x55f2c2231800 td/osd-mapper/2/block) close 2026-02-22T20:20:51.522 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bdev(0x55f2c2231800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-22T20:20:51.522 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bdev(0x55f2c2231800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:51.523 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T20:20:51.523 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bdev(0x55f2c2231c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-22T20:20:51.523 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bdev(0x55f2c2231c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:51.523 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2026-02-22T20:20:51.523 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bluefs mount 2026-02-22T20:20:51.523 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T20:20:51.526 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.526 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.526 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.526 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.526 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.526 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.518+0000 7fa75a5b3d40 1 bluefs mount shared_bdev_used = 0 2026-02-22T20:20:51.526 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.522+0000 7fa75a5b3d40 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T20:20:51.539 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.534+0000 7fa75a5b3d40 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T20:20:51.539 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.534+0000 7fa75a5b3d40 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2051 2026-02-22T20:20:51.539 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.534+0000 7fa75a5b3d40 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2026-02-22T20:20:51.539 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.534+0000 7fa75a5b3d40 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T20:20:51.539 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.534+0000 7fa75a5b3d40 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2026-02-22T20:20:51.539 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.534+0000 7fa75a5b3d40 1 freelist init 2026-02-22T20:20:51.539 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.534+0000 7fa75a5b3d40 1 freelist _read_cfg 2026-02-22T20:20:51.539 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.534+0000 7fa75a5b3d40 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-22T20:20:51.541 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.534+0000 7fa75a5b3d40 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 3 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc6000, fragmentation 7.62941e-08 2026-02-22T20:20:51.541 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.534+0000 7fa75a5b3d40 1 bluefs umount 2026-02-22T20:20:51.541 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.534+0000 7fa75a5b3d40 1 bdev(0x55f2c2231c00 td/osd-mapper/2/block) close 2026-02-22T20:20:51.582 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.578+0000 7fa75a5b3d40 1 bdev(0x55f2c2231c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-22T20:20:51.582 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.578+0000 7fa75a5b3d40 1 bdev(0x55f2c2231c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:51.582 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.578+0000 7fa75a5b3d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2026-02-22T20:20:51.582 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.578+0000 7fa75a5b3d40 1 bluefs mount 2026-02-22T20:20:51.582 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.578+0000 7fa75a5b3d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T20:20:51.584 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.578+0000 7fa75a5b3d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.584 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.578+0000 7fa75a5b3d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.584 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.578+0000 7fa75a5b3d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.584 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.578+0000 7fa75a5b3d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.584 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.578+0000 7fa75a5b3d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:51.584 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.578+0000 7fa75a5b3d40 1 bluefs mount shared_bdev_used = 27656192 2026-02-22T20:20:51.585 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.578+0000 7fa75a5b3d40 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T20:20:51.596 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.590+0000 7fa75a5b3d40 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T20:20:51.596 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.590+0000 7fa75a5b3d40 1 bluefs umount 2026-02-22T20:20:51.596 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.590+0000 7fa75a5b3d40 1 bdev(0x55f2c2231c00 td/osd-mapper/2/block) close 2026-02-22T20:20:51.634 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.630+0000 7fa75a5b3d40 1 freelist shutdown 2026-02-22T20:20:51.634 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:51.630+0000 7fa75a5b3d40 1 bdev(0x55f2c2231800 td/osd-mapper/2/block) close 2026-02-22T20:20:51.677 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:51.677 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:51.678 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:51.678 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:51.678 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:51.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-22T20:20:51.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:101: TEST_truncated_sna_record: grep -a SNA_ /tmp/oo2.dump 2026-02-22T20:20:51.679 INFO:tasks.workunit.client.0.trial150.stderr:grep: /tmp/oo2.dump: No such file or directory 2026-02-22T20:20:51.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-22T20:20:51.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:102: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2026-02-22T20:20:51.808 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-22T20:20:51.809 INFO:tasks.workunit.client.0.trial150.stdout:00000000 0b 05 58 02 00 00 77 a8 cc 62 b0 08 4f 99 be 84 |..X...w..b..O...| 2026-02-22T20:20:51.809 INFO:tasks.workunit.client.0.trial150.stdout:00000010 6b 5f 87 b5 e0 2b 02 00 00 00 19 00 00 00 00 00 |k_...+..........| 2026-02-22T20:20:51.809 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-22T20:20:51.809 INFO:tasks.workunit.client.0.trial150.stdout:* 2026-02-22T20:20:51.809 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-22T20:20:51.809 INFO:tasks.workunit.client.0.trial150.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-22T20:20:51.809 INFO:tasks.workunit.client.0.trial150.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-22T20:20:51.809 INFO:tasks.workunit.client.0.trial150.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-22T20:20:51.809 INFO:tasks.workunit.client.0.trial150.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-22T20:20:51.809 INFO:tasks.workunit.client.0.trial150.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-22T20:20:51.810 INFO:tasks.workunit.client.0.trial150.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:00000220 72 65 19 00 00 00 0d 00 00 00 b8 b1 4a 6f e8 5a |re..........Jo.Z| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:00000230 4d 06 ad b1 6d a1 3b 14 be 92 00 00 00 00 00 00 |M...m.;.........| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:00000240 00 00 19 00 00 00 13 65 9b 69 62 e9 a0 0b 01 00 |.......e.ib.....| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:00000250 00 00 01 00 00 00 01 00 00 00 19 00 00 00 |..............| 2026-02-22T20:20:51.811 INFO:tasks.workunit.client.0.trial150.stdout:0000025e 2026-02-22T20:20:51.812 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.812 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-22T20:20:51.812 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.812 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:51.812 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-22T20:20:51.812 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:51.812 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:51.812 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-22T20:20:51.812 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-22T20:20:51.812 INFO:tasks.workunit.client.0.trial150.stdout:00000047 2026-02-22T20:20:51.812 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.812 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout:0000003f 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout:00000047 2026-02-22T20:20:51.813 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:51.814 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:51.815 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.815 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:51.815 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:51.815 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:51.815 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:51.815 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:51.815 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:51.815 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.815 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:51.815 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:51.815 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-22T20:20:51.816 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.817 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-22T20:20:51.817 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:51.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-22T20:20:51.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2026-02-22T20:20:51.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-22T20:20:51.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: wc -l 2026-02-22T20:20:52.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:104: TEST_truncated_sna_record: local num_sna_b4=5 2026-02-22T20:20:52.147 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: expr 3 - 1 2026-02-22T20:20:52.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: seq 0 2 2026-02-22T20:20:52.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-22T20:20:52.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/0 2026-02-22T20:20:52.150 INFO:tasks.workunit.client.0.trial150.stdout:corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0) 2026-02-22T20:20:52.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.0 (db: td/osd-mapper/0)' 2026-02-22T20:20:52.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2026-02-22T20:20:52.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2026-02-22T20:20:52.170 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.166+0000 7f2279cdcd40 1 bdev(0x562478533800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-22T20:20:52.170 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.166+0000 7f2279cdcd40 1 bdev(0x562478533800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:52.170 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.166+0000 7f2279cdcd40 1 bdev(0x562478533800 td/osd-mapper/0/block) close 2026-02-22T20:20:52.238 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.234+0000 7f2279cdcd40 1 bdev(0x562478533800 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-22T20:20:52.238 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.234+0000 7f2279cdcd40 1 bdev(0x562478533800 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:52.238 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.234+0000 7f2279cdcd40 1 bluestore(td/osd-mapper/0) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T20:20:52.239 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.234+0000 7f2279cdcd40 1 bdev(0x562478533c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-22T20:20:52.239 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.234+0000 7f2279cdcd40 1 bdev(0x562478533c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:52.239 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.234+0000 7f2279cdcd40 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2026-02-22T20:20:52.239 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.234+0000 7f2279cdcd40 1 bluefs mount 2026-02-22T20:20:52.240 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.234+0000 7f2279cdcd40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T20:20:52.242 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.238+0000 7f2279cdcd40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:52.242 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.238+0000 7f2279cdcd40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:52.242 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.238+0000 7f2279cdcd40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:52.242 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.238+0000 7f2279cdcd40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:52.242 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.238+0000 7f2279cdcd40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:52.243 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.238+0000 7f2279cdcd40 1 bluefs mount shared_bdev_used = 0 2026-02-22T20:20:52.243 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.238+0000 7f2279cdcd40 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T20:20:52.255 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.250+0000 7f2279cdcd40 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T20:20:52.255 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.250+0000 7f2279cdcd40 1 bluestore(td/osd-mapper/0) _open_super_meta old nid_max 2050 2026-02-22T20:20:52.256 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.250+0000 7f2279cdcd40 1 bluestore(td/osd-mapper/0) _open_super_meta old blobid_max 20480 2026-02-22T20:20:52.256 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.250+0000 7f2279cdcd40 1 bluestore(td/osd-mapper/0) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T20:20:52.256 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.250+0000 7f2279cdcd40 1 bluestore(td/osd-mapper/0) _open_super_meta min_alloc_size 0x1000 2026-02-22T20:20:52.256 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.250+0000 7f2279cdcd40 1 freelist init 2026-02-22T20:20:52.256 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.250+0000 7f2279cdcd40 1 freelist _read_cfg 2026-02-22T20:20:52.256 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.250+0000 7f2279cdcd40 1 bluestore(td/osd-mapper/0) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-22T20:20:52.257 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.250+0000 7f2279cdcd40 1 bluestore(td/osd-mapper/0) _init_alloc loaded 100 GiB in 3 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc8000, fragmentation 7.62941e-08 2026-02-22T20:20:52.257 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.250+0000 7f2279cdcd40 1 bluefs umount 2026-02-22T20:20:52.257 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.250+0000 7f2279cdcd40 1 bdev(0x562478533c00 td/osd-mapper/0/block) close 2026-02-22T20:20:52.302 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.298+0000 7f2279cdcd40 1 bdev(0x562478533c00 td/osd-mapper/0/block) open path td/osd-mapper/0/block 2026-02-22T20:20:52.302 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.298+0000 7f2279cdcd40 1 bdev(0x562478533c00 td/osd-mapper/0/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:52.302 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.298+0000 7f2279cdcd40 1 bluefs add_block_device bdev 1 path td/osd-mapper/0/block size 100 GiB 2026-02-22T20:20:52.303 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.298+0000 7f2279cdcd40 1 bluefs mount 2026-02-22T20:20:52.303 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.298+0000 7f2279cdcd40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T20:20:52.304 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.298+0000 7f2279cdcd40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:52.304 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.298+0000 7f2279cdcd40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:52.304 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.298+0000 7f2279cdcd40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:52.304 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.298+0000 7f2279cdcd40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:52.304 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.298+0000 7f2279cdcd40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:52.304 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.298+0000 7f2279cdcd40 1 bluefs mount shared_bdev_used = 27656192 2026-02-22T20:20:52.304 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.298+0000 7f2279cdcd40 1 bluestore(td/osd-mapper/0) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T20:20:52.315 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.310+0000 7f2279cdcd40 1 bluestore(td/osd-mapper/0) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T20:20:52.316 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-22T20:20:52.316 INFO:tasks.workunit.client.0.trial150.stdout:00000000 0b 05 58 02 00 00 77 a8 cc 62 b0 08 4f 99 be 84 |..X...w..b..O...| 2026-02-22T20:20:52.317 INFO:tasks.workunit.client.0.trial150.stdout:00000010 6b 5f 87 b5 e0 2b 00 00 00 00 18 00 00 00 00 00 |k_...+..........| 2026-02-22T20:20:52.317 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-22T20:20:52.317 INFO:tasks.workunit.client.0.trial150.stdout:* 2026-02-22T20:20:52.317 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-22T20:20:52.317 INFO:tasks.workunit.client.0.trial150.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-22T20:20:52.317 INFO:tasks.workunit.client.0.trial150.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-22T20:20:52.317 INFO:tasks.workunit.client.0.trial150.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-22T20:20:52.317 INFO:tasks.workunit.client.0.trial150.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-22T20:20:52.317 INFO:tasks.workunit.client.0.trial150.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-22T20:20:52.318 INFO:tasks.workunit.client.0.trial150.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-22T20:20:52.318 INFO:tasks.workunit.client.0.trial150.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-22T20:20:52.318 INFO:tasks.workunit.client.0.trial150.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-22T20:20:52.318 INFO:tasks.workunit.client.0.trial150.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-22T20:20:52.318 INFO:tasks.workunit.client.0.trial150.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-22T20:20:52.318 INFO:tasks.workunit.client.0.trial150.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-22T20:20:52.318 INFO:tasks.workunit.client.0.trial150.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-22T20:20:52.318 INFO:tasks.workunit.client.0.trial150.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-22T20:20:52.318 INFO:tasks.workunit.client.0.trial150.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-22T20:20:52.318 INFO:tasks.workunit.client.0.trial150.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-22T20:20:52.318 INFO:tasks.workunit.client.0.trial150.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-22T20:20:52.318 INFO:tasks.workunit.client.0.trial150.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-22T20:20:52.319 INFO:tasks.workunit.client.0.trial150.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-22T20:20:52.319 INFO:tasks.workunit.client.0.trial150.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-22T20:20:52.319 INFO:tasks.workunit.client.0.trial150.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-22T20:20:52.319 INFO:tasks.workunit.client.0.trial150.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-22T20:20:52.319 INFO:tasks.workunit.client.0.trial150.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-22T20:20:52.319 INFO:tasks.workunit.client.0.trial150.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-22T20:20:52.319 INFO:tasks.workunit.client.0.trial150.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-22T20:20:52.319 INFO:tasks.workunit.client.0.trial150.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-22T20:20:52.319 INFO:tasks.workunit.client.0.trial150.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-22T20:20:52.319 INFO:tasks.workunit.client.0.trial150.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-22T20:20:52.319 INFO:tasks.workunit.client.0.trial150.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-22T20:20:52.319 INFO:tasks.workunit.client.0.trial150.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-22T20:20:52.320 INFO:tasks.workunit.client.0.trial150.stdout:00000220 72 65 18 00 00 00 05 00 00 00 17 90 89 d4 c7 43 |re.............C| 2026-02-22T20:20:52.320 INFO:tasks.workunit.client.0.trial150.stdout:00000230 48 1a 96 2e b2 f0 09 b9 b3 84 00 00 00 00 00 00 |H...............| 2026-02-22T20:20:52.320 INFO:tasks.workunit.client.0.trial150.stdout:00000240 00 00 18 00 00 00 0b 65 9b 69 8e 81 98 2e 01 00 |.......e.i......| 2026-02-22T20:20:52.320 INFO:tasks.workunit.client.0.trial150.stdout:00000250 00 00 01 00 00 00 01 00 00 00 18 00 00 00 |..............| 2026-02-22T20:20:52.320 INFO:tasks.workunit.client.0.trial150.stdout:0000025e 2026-02-22T20:20:52.320 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:52.320 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-22T20:20:52.320 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:52.320 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:52.320 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-22T20:20:52.320 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:52.320 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:52.321 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-22T20:20:52.321 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-22T20:20:52.321 INFO:tasks.workunit.client.0.trial150.stdout:00000047 2026-02-22T20:20:52.321 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:52.321 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:52.321 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-22T20:20:52.321 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:52.321 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:52.321 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-22T20:20:52.321 INFO:tasks.workunit.client.0.trial150.stdout:0000003f 2026-02-22T20:20:52.321 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout:00000047 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:52.322 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:52.323 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:52.323 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:52.323 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:52.323 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:52.323 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:52.323 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:52.323 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:52.323 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:52.323 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:52.323 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:52.323 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:52.323 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:52.324 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:52.324 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:52.324 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:52.324 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:52.324 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:52.324 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:52.324 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:52.324 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:52.324 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:52.324 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:52.324 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:52.324 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:52.325 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:52.325 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:52.325 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:52.325 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:52.325 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:52.325 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:52.325 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-22T20:20:52.325 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:52.325 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-22T20:20:52.325 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:52.326 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.310+0000 7f2279cdcd40 1 bluefs umount 2026-02-22T20:20:52.326 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.310+0000 7f2279cdcd40 1 bdev(0x562478533c00 td/osd-mapper/0/block) close 2026-02-22T20:20:52.362 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.358+0000 7f2279cdcd40 1 freelist shutdown 2026-02-22T20:20:52.362 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:52.358+0000 7f2279cdcd40 1 bdev(0x562478533800 td/osd-mapper/0/block) close 2026-02-22T20:20:52.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 dump p 2026-02-22T20:20:52.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2026-02-22T20:20:52.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-22T20:20:52.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:52.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:20:52.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2026-02-22T20:20:52.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2026-02-22T20:20:52.635 INFO:tasks.workunit.client.0.trial150.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:52.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2026-02-22T20:20:52.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.bSv6r0jlLv_the_val 2026-02-22T20:20:52.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:20:52.637 INFO:tasks.workunit.client.0.trial150.stdout:Value dumped in: /tmp/tmp.bSv6r0jlLv_the_val 2026-02-22T20:20:52.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.bSv6r0jlLv_the_val' 2026-02-22T20:20:52.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.bSv6r0jlLv_the_val 2026-02-22T20:20:52.771 INFO:tasks.workunit.client.0.trial150.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2026-02-22T20:20:52.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-22T20:20:52.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.bSv6r0jlLv_the_val 2026-02-22T20:20:52.884 INFO:tasks.workunit.client.0.trial150.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2026-02-22T20:20:52.885 INFO:tasks.workunit.client.0.trial150.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2026-02-22T20:20:52.885 INFO:tasks.workunit.client.0.trial150.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2026-02-22T20:20:52.885 INFO:tasks.workunit.client.0.trial150.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2026-02-22T20:20:52.885 INFO:tasks.workunit.client.0.trial150.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2026-02-22T20:20:52.886 INFO:tasks.workunit.client.0.trial150.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2026-02-22T20:20:52.886 INFO:tasks.workunit.client.0.trial150.stdout:0000060 0000 0100 0000 0000 0000 0000 2026-02-22T20:20:52.886 INFO:tasks.workunit.client.0.trial150.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2026-02-22T20:20:52.886 INFO:tasks.workunit.client.0.trial150.stdout:0000073 2026-02-22T20:20:52.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2026-02-22T20:20:52.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:53.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/0 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.bSv6r0jlLv_the_val 2026-02-22T20:20:53.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:20:53.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-22T20:20:53.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2026-02-22T20:20:53.333 INFO:tasks.workunit.client.0.trial150.stdout:corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1) 2026-02-22T20:20:53.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.1 (db: td/osd-mapper/1)' 2026-02-22T20:20:53.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2026-02-22T20:20:53.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-22T20:20:53.345 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.338+0000 7f5bd1d42d40 1 bdev(0x55eda898f800 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2026-02-22T20:20:53.345 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.338+0000 7f5bd1d42d40 1 bdev(0x55eda898f800 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:53.345 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.338+0000 7f5bd1d42d40 1 bdev(0x55eda898f800 td/osd-mapper/1/block) close 2026-02-22T20:20:53.393 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.390+0000 7f5bd1d42d40 1 bdev(0x55eda898f800 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2026-02-22T20:20:53.394 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.390+0000 7f5bd1d42d40 1 bdev(0x55eda898f800 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:53.394 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.390+0000 7f5bd1d42d40 1 bluestore(td/osd-mapper/1) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T20:20:53.394 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.390+0000 7f5bd1d42d40 1 bdev(0x55eda898fc00 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2026-02-22T20:20:53.394 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.390+0000 7f5bd1d42d40 1 bdev(0x55eda898fc00 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:53.394 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.390+0000 7f5bd1d42d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2026-02-22T20:20:53.394 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.390+0000 7f5bd1d42d40 1 bluefs mount 2026-02-22T20:20:53.394 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.390+0000 7f5bd1d42d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T20:20:53.398 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.394+0000 7f5bd1d42d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:53.398 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.394+0000 7f5bd1d42d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:53.398 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.394+0000 7f5bd1d42d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:53.398 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.394+0000 7f5bd1d42d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:53.398 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.394+0000 7f5bd1d42d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:53.398 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.394+0000 7f5bd1d42d40 1 bluefs mount shared_bdev_used = 0 2026-02-22T20:20:53.398 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.394+0000 7f5bd1d42d40 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T20:20:53.410 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.406+0000 7f5bd1d42d40 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T20:20:53.411 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.406+0000 7f5bd1d42d40 1 bluestore(td/osd-mapper/1) _open_super_meta old nid_max 2051 2026-02-22T20:20:53.411 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.406+0000 7f5bd1d42d40 1 bluestore(td/osd-mapper/1) _open_super_meta old blobid_max 20480 2026-02-22T20:20:53.411 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.406+0000 7f5bd1d42d40 1 bluestore(td/osd-mapper/1) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T20:20:53.411 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.406+0000 7f5bd1d42d40 1 bluestore(td/osd-mapper/1) _open_super_meta min_alloc_size 0x1000 2026-02-22T20:20:53.411 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.406+0000 7f5bd1d42d40 1 freelist init 2026-02-22T20:20:53.411 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.406+0000 7f5bd1d42d40 1 freelist _read_cfg 2026-02-22T20:20:53.411 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.406+0000 7f5bd1d42d40 1 bluestore(td/osd-mapper/1) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-22T20:20:53.412 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.406+0000 7f5bd1d42d40 1 bluestore(td/osd-mapper/1) _init_alloc loaded 100 GiB in 3 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc8000, fragmentation 7.62941e-08 2026-02-22T20:20:53.412 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.406+0000 7f5bd1d42d40 1 bluefs umount 2026-02-22T20:20:53.412 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.406+0000 7f5bd1d42d40 1 bdev(0x55eda898fc00 td/osd-mapper/1/block) close 2026-02-22T20:20:53.450 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.446+0000 7f5bd1d42d40 1 bdev(0x55eda898fc00 td/osd-mapper/1/block) open path td/osd-mapper/1/block 2026-02-22T20:20:53.450 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.446+0000 7f5bd1d42d40 1 bdev(0x55eda898fc00 td/osd-mapper/1/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:53.450 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.446+0000 7f5bd1d42d40 1 bluefs add_block_device bdev 1 path td/osd-mapper/1/block size 100 GiB 2026-02-22T20:20:53.450 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.446+0000 7f5bd1d42d40 1 bluefs mount 2026-02-22T20:20:53.450 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.446+0000 7f5bd1d42d40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T20:20:53.452 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.446+0000 7f5bd1d42d40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:53.452 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.446+0000 7f5bd1d42d40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:53.452 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.446+0000 7f5bd1d42d40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:53.452 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.446+0000 7f5bd1d42d40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:53.452 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.446+0000 7f5bd1d42d40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:53.452 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.446+0000 7f5bd1d42d40 1 bluefs mount shared_bdev_used = 27656192 2026-02-22T20:20:53.452 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.446+0000 7f5bd1d42d40 1 bluestore(td/osd-mapper/1) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T20:20:53.465 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.458+0000 7f5bd1d42d40 1 bluestore(td/osd-mapper/1) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T20:20:53.466 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-22T20:20:53.466 INFO:tasks.workunit.client.0.trial150.stdout:00000000 0b 05 58 02 00 00 77 a8 cc 62 b0 08 4f 99 be 84 |..X...w..b..O...| 2026-02-22T20:20:53.466 INFO:tasks.workunit.client.0.trial150.stdout:00000010 6b 5f 87 b5 e0 2b 01 00 00 00 18 00 00 00 00 00 |k_...+..........| 2026-02-22T20:20:53.466 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-22T20:20:53.466 INFO:tasks.workunit.client.0.trial150.stdout:* 2026-02-22T20:20:53.466 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-22T20:20:53.466 INFO:tasks.workunit.client.0.trial150.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-22T20:20:53.466 INFO:tasks.workunit.client.0.trial150.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-22T20:20:53.467 INFO:tasks.workunit.client.0.trial150.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-22T20:20:53.467 INFO:tasks.workunit.client.0.trial150.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-22T20:20:53.467 INFO:tasks.workunit.client.0.trial150.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-22T20:20:53.467 INFO:tasks.workunit.client.0.trial150.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-22T20:20:53.467 INFO:tasks.workunit.client.0.trial150.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-22T20:20:53.467 INFO:tasks.workunit.client.0.trial150.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-22T20:20:53.467 INFO:tasks.workunit.client.0.trial150.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-22T20:20:53.467 INFO:tasks.workunit.client.0.trial150.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-22T20:20:53.467 INFO:tasks.workunit.client.0.trial150.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-22T20:20:53.467 INFO:tasks.workunit.client.0.trial150.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-22T20:20:53.467 INFO:tasks.workunit.client.0.trial150.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-22T20:20:53.467 INFO:tasks.workunit.client.0.trial150.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-22T20:20:53.468 INFO:tasks.workunit.client.0.trial150.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-22T20:20:53.468 INFO:tasks.workunit.client.0.trial150.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-22T20:20:53.468 INFO:tasks.workunit.client.0.trial150.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-22T20:20:53.468 INFO:tasks.workunit.client.0.trial150.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-22T20:20:53.468 INFO:tasks.workunit.client.0.trial150.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-22T20:20:53.468 INFO:tasks.workunit.client.0.trial150.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-22T20:20:53.468 INFO:tasks.workunit.client.0.trial150.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-22T20:20:53.468 INFO:tasks.workunit.client.0.trial150.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-22T20:20:53.468 INFO:tasks.workunit.client.0.trial150.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-22T20:20:53.468 INFO:tasks.workunit.client.0.trial150.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-22T20:20:53.468 INFO:tasks.workunit.client.0.trial150.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-22T20:20:53.468 INFO:tasks.workunit.client.0.trial150.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-22T20:20:53.469 INFO:tasks.workunit.client.0.trial150.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-22T20:20:53.469 INFO:tasks.workunit.client.0.trial150.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-22T20:20:53.469 INFO:tasks.workunit.client.0.trial150.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-22T20:20:53.469 INFO:tasks.workunit.client.0.trial150.stdout:00000220 72 65 18 00 00 00 09 00 00 00 22 8f df d9 fd 50 |re........"....P| 2026-02-22T20:20:53.469 INFO:tasks.workunit.client.0.trial150.stdout:00000230 46 e6 8a 9f 1b 12 9b 63 36 bc 00 00 00 00 00 00 |F......c6.......| 2026-02-22T20:20:53.469 INFO:tasks.workunit.client.0.trial150.stdout:00000240 00 00 18 00 00 00 0f 65 9b 69 77 f6 04 1a 01 00 |.......e.iw.....| 2026-02-22T20:20:53.469 INFO:tasks.workunit.client.0.trial150.stdout:00000250 00 00 01 00 00 00 01 00 00 00 18 00 00 00 |..............| 2026-02-22T20:20:53.469 INFO:tasks.workunit.client.0.trial150.stdout:0000025e 2026-02-22T20:20:53.469 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:53.469 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-22T20:20:53.469 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:53.469 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:53.470 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-22T20:20:53.470 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:53.470 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:53.470 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-22T20:20:53.470 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-22T20:20:53.470 INFO:tasks.workunit.client.0.trial150.stdout:00000047 2026-02-22T20:20:53.470 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:53.470 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:53.470 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-22T20:20:53.470 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:53.470 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:53.471 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-22T20:20:53.471 INFO:tasks.workunit.client.0.trial150.stdout:0000003f 2026-02-22T20:20:53.471 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:53.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:53.471 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-22T20:20:53.471 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:53.471 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:53.471 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-22T20:20:53.471 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-22T20:20:53.471 INFO:tasks.workunit.client.0.trial150.stdout:00000047 2026-02-22T20:20:53.471 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:53.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:53.472 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:53.472 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:53.472 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:53.472 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:53.472 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:53.472 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:53.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:53.472 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:53.472 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:53.472 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:53.472 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:53.472 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:53.473 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:53.473 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:53.473 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:53.473 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:53.473 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:53.473 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:53.473 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:53.473 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:53.473 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:53.473 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:53.473 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:53.473 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:53.474 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:53.474 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:53.474 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:53.474 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:53.474 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:53.474 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:53.474 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:53.474 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:53.474 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:53.474 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:53.474 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-22T20:20:53.474 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:53.475 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-22T20:20:53.475 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:53.475 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.458+0000 7f5bd1d42d40 1 bluefs umount 2026-02-22T20:20:53.475 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.458+0000 7f5bd1d42d40 1 bdev(0x55eda898fc00 td/osd-mapper/1/block) close 2026-02-22T20:20:53.514 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.510+0000 7f5bd1d42d40 1 freelist shutdown 2026-02-22T20:20:53.514 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:53.510+0000 7f5bd1d42d40 1 bdev(0x55eda898f800 td/osd-mapper/1/block) close 2026-02-22T20:20:53.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-22T20:20:53.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2026-02-22T20:20:53.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-22T20:20:53.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:53.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:20:53.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2026-02-22T20:20:53.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2026-02-22T20:20:53.803 INFO:tasks.workunit.client.0.trial150.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:53.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2026-02-22T20:20:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.wak6mbA1X8_the_val 2026-02-22T20:20:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:20:53.805 INFO:tasks.workunit.client.0.trial150.stdout:Value dumped in: /tmp/tmp.wak6mbA1X8_the_val 2026-02-22T20:20:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.wak6mbA1X8_the_val' 2026-02-22T20:20:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.wak6mbA1X8_the_val 2026-02-22T20:20:53.951 INFO:tasks.workunit.client.0.trial150.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2026-02-22T20:20:54.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-22T20:20:54.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.wak6mbA1X8_the_val 2026-02-22T20:20:54.058 INFO:tasks.workunit.client.0.trial150.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2026-02-22T20:20:54.058 INFO:tasks.workunit.client.0.trial150.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2026-02-22T20:20:54.058 INFO:tasks.workunit.client.0.trial150.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2026-02-22T20:20:54.058 INFO:tasks.workunit.client.0.trial150.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2026-02-22T20:20:54.058 INFO:tasks.workunit.client.0.trial150.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2026-02-22T20:20:54.058 INFO:tasks.workunit.client.0.trial150.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2026-02-22T20:20:54.058 INFO:tasks.workunit.client.0.trial150.stdout:0000060 0000 0100 0000 0000 0000 0000 2026-02-22T20:20:54.058 INFO:tasks.workunit.client.0.trial150.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2026-02-22T20:20:54.058 INFO:tasks.workunit.client.0.trial150.stdout:0000073 2026-02-22T20:20:54.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2026-02-22T20:20:54.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:54.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.wak6mbA1X8_the_val 2026-02-22T20:20:54.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:20:54.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:106: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-22T20:20:54.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:108: TEST_truncated_sna_record: kvdir=td/osd-mapper/2 2026-02-22T20:20:54.534 INFO:tasks.workunit.client.0.trial150.stdout:corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2) 2026-02-22T20:20:54.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:109: TEST_truncated_sna_record: echo 'corrupting the SnapMapper DB of osd.2 (db: td/osd-mapper/2)' 2026-02-22T20:20:54.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: (( extr_dbg >= 3 )) 2026-02-22T20:20:54.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:110: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2026-02-22T20:20:54.545 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.538+0000 7f8cbe1cbd40 1 bdev(0x55d1a7ba3800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-22T20:20:54.546 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.538+0000 7f8cbe1cbd40 1 bdev(0x55d1a7ba3800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:54.546 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.542+0000 7f8cbe1cbd40 1 bdev(0x55d1a7ba3800 td/osd-mapper/2/block) close 2026-02-22T20:20:54.605 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bdev(0x55d1a7ba3800 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-22T20:20:54.606 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bdev(0x55d1a7ba3800 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:54.606 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bluestore(td/osd-mapper/2) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 kv_onode 0.04 data 0.06 2026-02-22T20:20:54.606 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bdev(0x55d1a7ba3c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-22T20:20:54.606 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bdev(0x55d1a7ba3c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:54.606 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2026-02-22T20:20:54.606 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bluefs mount 2026-02-22T20:20:54.606 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T20:20:54.609 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:54.609 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:54.609 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:54.610 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:54.610 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:54.610 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.602+0000 7f8cbe1cbd40 1 bluefs mount shared_bdev_used = 0 2026-02-22T20:20:54.610 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.606+0000 7f8cbe1cbd40 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T20:20:54.623 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.618+0000 7f8cbe1cbd40 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T20:20:54.623 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.618+0000 7f8cbe1cbd40 1 bluestore(td/osd-mapper/2) _open_super_meta old nid_max 2051 2026-02-22T20:20:54.623 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.618+0000 7f8cbe1cbd40 1 bluestore(td/osd-mapper/2) _open_super_meta old blobid_max 20480 2026-02-22T20:20:54.623 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.618+0000 7f8cbe1cbd40 1 bluestore(td/osd-mapper/2) _open_super_meta ondisk_format 4 compat_ondisk_format 3 2026-02-22T20:20:54.623 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.618+0000 7f8cbe1cbd40 1 bluestore(td/osd-mapper/2) _open_super_meta min_alloc_size 0x1000 2026-02-22T20:20:54.623 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.618+0000 7f8cbe1cbd40 1 freelist init 2026-02-22T20:20:54.623 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.618+0000 7f8cbe1cbd40 1 freelist _read_cfg 2026-02-22T20:20:54.623 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.618+0000 7f8cbe1cbd40 1 bluestore(td/osd-mapper/2) _open_fm effective freelist_type = null, freelist_alloc_size = 0x1000, min_alloc_size = 0x1000 2026-02-22T20:20:54.624 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.618+0000 7f8cbe1cbd40 1 bluestore(td/osd-mapper/2) _init_alloc loaded 100 GiB in 3 extents, allocator type hybrid, capacity 0x1900000000, block size 0x1000, free 0x18fffc6000, fragmentation 7.62941e-08 2026-02-22T20:20:54.624 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.618+0000 7f8cbe1cbd40 1 bluefs umount 2026-02-22T20:20:54.624 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.618+0000 7f8cbe1cbd40 1 bdev(0x55d1a7ba3c00 td/osd-mapper/2/block) close 2026-02-22T20:20:54.674 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.670+0000 7f8cbe1cbd40 1 bdev(0x55d1a7ba3c00 td/osd-mapper/2/block) open path td/osd-mapper/2/block 2026-02-22T20:20:54.674 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.670+0000 7f8cbe1cbd40 1 bdev(0x55d1a7ba3c00 td/osd-mapper/2/block) open size 107374182400 (0x1900000000, 100 GiB) block_size 4096 (4 KiB) non-rotational device, discard supported 2026-02-22T20:20:54.674 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.670+0000 7f8cbe1cbd40 1 bluefs add_block_device bdev 1 path td/osd-mapper/2/block size 100 GiB 2026-02-22T20:20:54.674 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.670+0000 7f8cbe1cbd40 1 bluefs mount 2026-02-22T20:20:54.674 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.670+0000 7f8cbe1cbd40 1 bluefs _init_alloc shared, id 1, capacity 0x1900000000, block size 0x10000 2026-02-22T20:20:54.676 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.670+0000 7f8cbe1cbd40 1 bluefs mount final locked allocations 0 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:54.676 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.670+0000 7f8cbe1cbd40 1 bluefs mount final locked allocations 1 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:54.676 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.670+0000 7f8cbe1cbd40 1 bluefs mount final locked allocations 2 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:54.677 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.670+0000 7f8cbe1cbd40 1 bluefs mount final locked allocations 3 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:54.677 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.670+0000 7f8cbe1cbd40 1 bluefs mount final locked allocations 4 <0x0~0, [0x0~0], 0x0~0> => <0x0~0, [0x0~0], 0x0~0> 2026-02-22T20:20:54.677 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.670+0000 7f8cbe1cbd40 1 bluefs mount shared_bdev_used = 27656192 2026-02-22T20:20:54.677 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.670+0000 7f8cbe1cbd40 1 bluestore(td/osd-mapper/2) _prepare_db_environment set db_paths to db,102005473280 db.slow,102005473280 2026-02-22T20:20:54.688 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.682+0000 7f8cbe1cbd40 1 bluestore(td/osd-mapper/2) _open_db opened rocksdb path db options compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 2026-02-22T20:20:54.689 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01.osd_superblock 2026-02-22T20:20:54.689 INFO:tasks.workunit.client.0.trial150.stdout:00000000 0b 05 58 02 00 00 77 a8 cc 62 b0 08 4f 99 be 84 |..X...w..b..O...| 2026-02-22T20:20:54.689 INFO:tasks.workunit.client.0.trial150.stdout:00000010 6b 5f 87 b5 e0 2b 02 00 00 00 19 00 00 00 00 00 |k_...+..........| 2026-02-22T20:20:54.689 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 2026-02-22T20:20:54.689 INFO:tasks.workunit.client.0.trial150.stdout:* 2026-02-22T20:20:54.689 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 fe ff 03 00 00 00 00 00 11 00 |................| 2026-02-22T20:20:54.689 INFO:tasks.workunit.client.0.trial150.stdout:00000050 00 00 01 00 00 00 00 00 00 00 1a 00 00 00 69 6e |..............in| 2026-02-22T20:20:54.689 INFO:tasks.workunit.client.0.trial150.stdout:00000060 69 74 69 61 6c 20 66 65 61 74 75 72 65 20 73 65 |itial feature se| 2026-02-22T20:20:54.690 INFO:tasks.workunit.client.0.trial150.stdout:00000070 74 28 7e 76 2e 31 38 29 02 00 00 00 00 00 00 00 |t(~v.18)........| 2026-02-22T20:20:54.690 INFO:tasks.workunit.client.0.trial150.stdout:00000080 0d 00 00 00 70 67 69 6e 66 6f 20 6f 62 6a 65 63 |....pginfo objec| 2026-02-22T20:20:54.690 INFO:tasks.workunit.client.0.trial150.stdout:00000090 74 03 00 00 00 00 00 00 00 0e 00 00 00 6f 62 6a |t............obj| 2026-02-22T20:20:54.690 INFO:tasks.workunit.client.0.trial150.stdout:000000a0 65 63 74 20 6c 6f 63 61 74 6f 72 04 00 00 00 00 |ect locator.....| 2026-02-22T20:20:54.690 INFO:tasks.workunit.client.0.trial150.stdout:000000b0 00 00 00 10 00 00 00 6c 61 73 74 5f 65 70 6f 63 |.......last_epoc| 2026-02-22T20:20:54.690 INFO:tasks.workunit.client.0.trial150.stdout:000000c0 68 5f 63 6c 65 61 6e 05 00 00 00 00 00 00 00 0a |h_clean.........| 2026-02-22T20:20:54.690 INFO:tasks.workunit.client.0.trial150.stdout:000000d0 00 00 00 63 61 74 65 67 6f 72 69 65 73 06 00 00 |...categories...| 2026-02-22T20:20:54.690 INFO:tasks.workunit.client.0.trial150.stdout:000000e0 00 00 00 00 00 0b 00 00 00 68 6f 62 6a 65 63 74 |.........hobject| 2026-02-22T20:20:54.690 INFO:tasks.workunit.client.0.trial150.stdout:000000f0 70 6f 6f 6c 07 00 00 00 00 00 00 00 07 00 00 00 |pool............| 2026-02-22T20:20:54.690 INFO:tasks.workunit.client.0.trial150.stdout:00000100 62 69 67 69 6e 66 6f 08 00 00 00 00 00 00 00 0b |biginfo.........| 2026-02-22T20:20:54.690 INFO:tasks.workunit.client.0.trial150.stdout:00000110 00 00 00 6c 65 76 65 6c 64 62 69 6e 66 6f 09 00 |...leveldbinfo..| 2026-02-22T20:20:54.690 INFO:tasks.workunit.client.0.trial150.stdout:00000120 00 00 00 00 00 00 0a 00 00 00 6c 65 76 65 6c 64 |..........leveld| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:00000130 62 6c 6f 67 0a 00 00 00 00 00 00 00 0a 00 00 00 |blog............| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:00000140 73 6e 61 70 6d 61 70 70 65 72 0b 00 00 00 00 00 |snapmapper......| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:00000150 00 00 0f 00 00 00 73 68 61 72 64 65 64 20 6f 62 |......sharded ob| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:00000160 6a 65 63 74 73 0c 00 00 00 00 00 00 00 11 00 00 |jects...........| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:00000170 00 74 72 61 6e 73 61 63 74 69 6f 6e 20 68 69 6e |.transaction hin| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:00000180 74 73 0d 00 00 00 00 00 00 00 0e 00 00 00 70 67 |ts............pg| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:00000190 20 6d 65 74 61 20 6f 62 6a 65 63 74 0e 00 00 00 | meta object....| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:000001a0 00 00 00 00 14 00 00 00 65 78 70 6c 69 63 69 74 |........explicit| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:000001b0 20 6d 69 73 73 69 6e 67 20 73 65 74 0f 00 00 00 | missing set....| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:000001c0 00 00 00 00 10 00 00 00 66 61 73 74 69 6e 66 6f |........fastinfo| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:000001d0 20 70 67 20 61 74 74 72 10 00 00 00 00 00 00 00 | pg attr........| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:000001e0 16 00 00 00 64 65 6c 65 74 65 73 20 69 6e 20 6d |....deletes in m| 2026-02-22T20:20:54.691 INFO:tasks.workunit.client.0.trial150.stdout:000001f0 69 73 73 69 6e 67 20 73 65 74 11 00 00 00 00 00 |issing set......| 2026-02-22T20:20:54.692 INFO:tasks.workunit.client.0.trial150.stdout:00000200 00 00 1c 00 00 00 6e 65 77 20 73 6e 61 70 6d 61 |......new snapma| 2026-02-22T20:20:54.692 INFO:tasks.workunit.client.0.trial150.stdout:00000210 70 70 65 72 20 6b 65 79 20 73 74 72 75 63 74 75 |pper key structu| 2026-02-22T20:20:54.692 INFO:tasks.workunit.client.0.trial150.stdout:00000220 72 65 19 00 00 00 0d 00 00 00 b8 b1 4a 6f e8 5a |re..........Jo.Z| 2026-02-22T20:20:54.692 INFO:tasks.workunit.client.0.trial150.stdout:00000230 4d 06 ad b1 6d a1 3b 14 be 92 00 00 00 00 00 00 |M...m.;.........| 2026-02-22T20:20:54.692 INFO:tasks.workunit.client.0.trial150.stdout:00000240 00 00 19 00 00 00 13 65 9b 69 62 e9 a0 0b 01 00 |.......e.ib.....| 2026-02-22T20:20:54.692 INFO:tasks.workunit.client.0.trial150.stdout:00000250 00 00 01 00 00 00 01 00 00 00 19 00 00 00 |..............| 2026-02-22T20:20:54.692 INFO:tasks.workunit.client.0.trial150.stdout:0000025e 2026-02-22T20:20:54.692 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:54.692 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%7b%3fC%c4%00%00%00%00%00%00%00%01~ 2026-02-22T20:20:54.692 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:54.692 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:54.693 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-22T20:20:54.693 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 02 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:54.693 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:54.693 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 02 |................| 2026-02-22T20:20:54.693 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-22T20:20:54.693 INFO:tasks.workunit.client.0.trial150.stdout:00000047 2026-02-22T20:20:54.693 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:54.693 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:54.693 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 39 00 00 00 04 03 27 00 00 00 00 00 00 00 |..9.....'.......| 2026-02-22T20:20:54.693 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 03 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:54.693 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 03 00 00 00 00 00 00 00 |...............| 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout:0000003f 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.OBJ_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 41 00 00 00 04 03 27 00 00 00 00 00 00 00 |..A.....'.......| 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout:00000010 06 00 00 00 6f 62 6a 78 78 78 05 00 00 00 00 00 |....objxxx......| 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout:00000020 00 00 af 75 d0 85 00 00 00 00 00 01 00 00 00 00 |...u............| 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 02 00 00 00 04 00 00 00 00 00 00 00 05 |................| 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout:00000040 00 00 00 00 00 00 00 |.......| 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout:00000047 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:54.694 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 01 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:54.695 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:54.695 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:54.695 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:54.695 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:54.695 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:54.695 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:20:54.695 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 02 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:54.695 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:54.695 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 02 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:54.695 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:54.695 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:54.695 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:54.696 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:54.696 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 03 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:54.696 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:54.696 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 03 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:54.696 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:54.696 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:54.696 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:54.696 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:54.696 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 04 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:54.696 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:54.696 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:54.696 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:54.697 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:54.697 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:54.697 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:20:54.697 INFO:tasks.workunit.client.0.trial150.stdout:00000000 01 01 35 00 00 00 05 00 00 00 00 00 00 00 04 03 |..5.............| 2026-02-22T20:20:54.697 INFO:tasks.workunit.client.0.trial150.stdout:00000010 27 00 00 00 00 00 00 00 06 00 00 00 6f 62 6a 78 |'...........objx| 2026-02-22T20:20:54.697 INFO:tasks.workunit.client.0.trial150.stdout:00000020 78 78 05 00 00 00 00 00 00 00 af 75 d0 85 00 00 |xx.........u....| 2026-02-22T20:20:54.697 INFO:tasks.workunit.client.0.trial150.stdout:00000030 00 00 00 01 00 00 00 00 00 00 00 |...........| 2026-02-22T20:20:54.697 INFO:tasks.workunit.client.0.trial150.stdout:0000003b 2026-02-22T20:20:54.697 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:54.697 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02~ 2026-02-22T20:20:54.697 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:54.697 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c1%a3%fcn%00%00%00%00%00%00%04%03~ 2026-02-22T20:20:54.698 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:20:54.698 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.682+0000 7f8cbe1cbd40 1 bluefs umount 2026-02-22T20:20:54.698 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.682+0000 7f8cbe1cbd40 1 bdev(0x55d1a7ba3c00 td/osd-mapper/2/block) close 2026-02-22T20:20:54.733 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.730+0000 7f8cbe1cbd40 1 freelist shutdown 2026-02-22T20:20:54.734 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:54.730+0000 7f8cbe1cbd40 1 bdev(0x55d1a7ba3800 td/osd-mapper/2/block) close 2026-02-22T20:20:54.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 dump p 2026-02-22T20:20:54.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_0000000000000003_000000000000000' 2026-02-22T20:20:54.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-22T20:20:55.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:114: TEST_truncated_sna_record: KY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:55.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:20:55.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: echo 'SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..' 2026-02-22T20:20:55.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:115: TEST_truncated_sna_record: cat -v 2026-02-22T20:20:55.059 INFO:tasks.workunit.client.0.trial150.stdout:SNA key: %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:55.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: mktemp -p /tmp --suffix=_the_val 2026-02-22T20:20:55.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:117: TEST_truncated_sna_record: tmp_fn1=/tmp/tmp.4SRiOby7sa_the_val 2026-02-22T20:20:55.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:20:55.061 INFO:tasks.workunit.client.0.trial150.stdout:Value dumped in: /tmp/tmp.4SRiOby7sa_the_val 2026-02-22T20:20:55.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:118: TEST_truncated_sna_record: echo 'Value dumped in: /tmp/tmp.4SRiOby7sa_the_val' 2026-02-22T20:20:55.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:119: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 get p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. out /tmp/tmp.4SRiOby7sa_the_val 2026-02-22T20:20:55.207 INFO:tasks.workunit.client.0.trial150.stdout:(p, %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx..) 2026-02-22T20:20:55.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-22T20:20:55.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:120: TEST_truncated_sna_record: od -xc /tmp/tmp.4SRiOby7sa_the_val 2026-02-22T20:20:55.314 INFO:tasks.workunit.client.0.trial150.stdout:0000000 0101 0035 0000 0003 0000 0000 0000 0304 2026-02-22T20:20:55.314 INFO:tasks.workunit.client.0.trial150.stdout: 001 001 5 \0 \0 \0 003 \0 \0 \0 \0 \0 \0 \0 004 003 2026-02-22T20:20:55.314 INFO:tasks.workunit.client.0.trial150.stdout:0000020 0027 0000 0000 0000 0006 0000 626f 786a 2026-02-22T20:20:55.315 INFO:tasks.workunit.client.0.trial150.stdout: ' \0 \0 \0 \0 \0 \0 \0 006 \0 \0 \0 o b j x 2026-02-22T20:20:55.315 INFO:tasks.workunit.client.0.trial150.stdout:0000040 7878 0003 0000 0000 0000 75af 85d0 0000 2026-02-22T20:20:55.315 INFO:tasks.workunit.client.0.trial150.stdout: x x 003 \0 \0 \0 \0 \0 \0 \0 257 u 320 205 \0 \0 2026-02-22T20:20:55.315 INFO:tasks.workunit.client.0.trial150.stdout:0000060 0000 0100 0000 0000 0000 0000 2026-02-22T20:20:55.315 INFO:tasks.workunit.client.0.trial150.stdout: \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0 2026-02-22T20:20:55.315 INFO:tasks.workunit.client.0.trial150.stdout:0000073 2026-02-22T20:20:55.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:122: TEST_truncated_sna_record: NKY=%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2026-02-22T20:20:55.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:123: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 rm p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:20:55.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:124: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/2 set p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 in /tmp/tmp.4SRiOby7sa_the_val 2026-02-22T20:20:55.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:126: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:20:55.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:129: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 ' 2026-02-22T20:20:55.782 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: echo --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 2026-02-22T20:20:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:130: TEST_truncated_sna_record: orig_osd_args=' --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-22T20:20:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-22T20:20:55.783 INFO:tasks.workunit.client.0.trial150.stdout:Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3/ 2026-02-22T20:20:55.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:131: TEST_truncated_sna_record: echo 'Copied OSD args: / --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3/ /--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3/' 2026-02-22T20:20:55.784 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: expr 3 - 1 2026-02-22T20:20:55.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: seq 0 2 2026-02-22T20:20:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-22T20:20:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=77a8cc62-b008-4f99-be84-6b5f87b5e02b --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-22T20:20:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 0 2026-02-22T20:20:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2026-02-22T20:20:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:20:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:20:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:20:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/0 2026-02-22T20:20:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=77a8cc62-b008-4f99-be84-6b5f87b5e02b --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-22T20:20:55.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:20:55.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:20:55.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:20:55.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/0' 2026-02-22T20:20:55.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/0/journal' 2026-02-22T20:20:55.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:20:55.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:20:55.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2026-02-22T20:20:55.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:20:55.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:20:55.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12039 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12039/$cluster-$name.asok' 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12039/$cluster-$name.asok' 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:20:55.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:20:55.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:20:55.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/0 2026-02-22T20:20:55.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:20:55.791 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:20:55.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=77a8cc62-b008-4f99-be84-6b5f87b5e02b --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/0 --osd-journal=td/osd-mapper/0/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.12039/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:20:55.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/0/whoami 2026-02-22T20:20:55.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:20:55.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:20:55.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:20:55.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:20:55.821 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:55.814+0000 7fe8b2468a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:55.822 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:55.818+0000 7fe8b2468a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:55.823 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:55.818+0000 7fe8b2468a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:55.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:20:55.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:20:55.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:20:55.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:20:55.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:20:55.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:20:55.955 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:20:55.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:20:55.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:20:55.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:20:56.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:20:56.215 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:56.210+0000 7fe8b2468a40 -1 Falling back to public interface 2026-02-22T20:20:56.838 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:56.830+0000 7fe8b2468a40 -1 osd.0 24 log_to_monitors true 2026-02-22T20:20:57.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:20:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:20:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:20:57.130 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:20:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:20:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:20:57.297 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 28 up_thru 28 down_at 25 last_clean_interval [5,24) [v2:127.0.0.1:6802/135799324,v1:127.0.0.1:6803/135799324] [v2:127.0.0.1:6804/135799324,v1:127.0.0.1:6805/135799324] exists,up 179089d4-c743-481a-962e-b2f009b9b384 2026-02-22T20:20:57.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:20:57.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:20:57.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:20:57.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-22T20:20:57.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=77a8cc62-b008-4f99-be84-6b5f87b5e02b --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-22T20:20:57.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 1 2026-02-22T20:20:57.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2026-02-22T20:20:57.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:20:57.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:20:57.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:20:57.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/1 2026-02-22T20:20:57.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=77a8cc62-b008-4f99-be84-6b5f87b5e02b --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-22T20:20:57.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:20:57.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:20:57.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:20:57.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/1' 2026-02-22T20:20:57.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/1/journal' 2026-02-22T20:20:57.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12039 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12039/$cluster-$name.asok' 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12039/$cluster-$name.asok' 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2026-02-22T20:20:57.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2026-02-22T20:20:57.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:20:57.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:20:57.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:20:57.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:20:57.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:20:57.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:20:57.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/1 2026-02-22T20:20:57.304 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:20:57.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:20:57.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=77a8cc62-b008-4f99-be84-6b5f87b5e02b --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/1 --osd-journal=td/osd-mapper/1/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.12039/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:20:57.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/1/whoami 2026-02-22T20:20:57.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:20:57.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:20:57.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:20:57.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:20:57.333 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:57.326+0000 7f33dd0e7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:57.335 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:57.330+0000 7f33dd0e7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:57.336 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:57.330+0000 7f33dd0e7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:57.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:20:57.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:20:57.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:20:57.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:20:57.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:20:57.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:20:57.466 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:20:57.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:20:57.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:20:57.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:20:57.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:20:57.715 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:57.710+0000 7f33dd0e7a40 -1 Falling back to public interface 2026-02-22T20:20:58.354 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:58.350+0000 7f33dd0e7a40 -1 osd.1 24 log_to_monitors true 2026-02-22T20:20:58.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:20:58.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:20:58.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:20:58.640 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:20:58.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:20:58.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:20:58.810 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 31 up_thru 31 down_at 25 last_clean_interval [9,24) [v2:127.0.0.1:6810/3984734842,v1:127.0.0.1:6811/3984734842] [v2:127.0.0.1:6812/3984734842,v1:127.0.0.1:6813/3984734842] exists,up 228fdfd9-fd50-46e6-8a9f-1b129b6336bc 2026-02-22T20:20:58.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:20:58.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:20:58.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:20:58.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:132: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: CEPH_ARGS='--fsid=77a8cc62-b008-4f99-be84-6b5f87b5e02b --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:134: TEST_truncated_sna_record: activate_osd td/osd-mapper 2 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-mapper 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-mapper/2 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=77a8cc62-b008-4f99-be84-6b5f87b5e02b --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3' 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-mapper/2' 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-mapper/2/journal' 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-mapper' 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:20:58.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:20:58.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:20:58.813 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:20:58.813 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:20:58.813 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12039 2026-02-22T20:20:58.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.12039/$cluster-$name.asok' 2026-02-22T20:20:58.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.12039/$cluster-$name.asok' 2026-02-22T20:20:58.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:20:58.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-mapper/$name.log' 2026-02-22T20:20:58.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-mapper/$name.pid' 2026-02-22T20:20:58.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:20:58.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:20:58.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:20:58.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:20:58.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:20:58.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:20:58.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-mapper/2 2026-02-22T20:20:58.814 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:20:58.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:20:58.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=77a8cc62-b008-4f99-be84-6b5f87b5e02b --auth-supported=none --mon-host=127.0.0.1:7144 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_pool_default_pg_autoscale_mode=off --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=5 --osd_scrub_retry_pg_state=5 --osd_scrub_retry_delay=3 --osd_pool_default_size=3 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-mapper/2 --osd-journal=td/osd-mapper/2/journal --chdir= --run-dir=td/osd-mapper '--admin-socket=/tmp/ceph-asok.12039/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-mapper/$name.log' '--pid-file=td/osd-mapper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:20:58.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-mapper/2/whoami 2026-02-22T20:20:58.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:20:58.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:20:58.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:20:58.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:20:58.841 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:58.834+0000 7fe82faf8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:58.843 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:58.838+0000 7fe82faf8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:58.844 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:58.838+0000 7fe82faf8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:20:58.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:20:58.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:20:58.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:20:58.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:20:58.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:20:58.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:20:58.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:20:58.985 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:20:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:20:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:20:59.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:20:59.299 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:59.294+0000 7fe82faf8a40 -1 Falling back to public interface 2026-02-22T20:21:00.001 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:20:59.994+0000 7fe82faf8a40 -1 osd.2 25 log_to_monitors true 2026-02-22T20:21:00.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:21:00.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:21:00.155 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:21:00.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:21:00.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:21:00.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:21:00.321 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 35 up_thru 35 down_at 25 last_clean_interval [13,25) [v2:127.0.0.1:6818/3796771404,v1:127.0.0.1:6819/3796771404] [v2:127.0.0.1:6820/3796771404,v1:127.0.0.1:6821/3796771404] exists,up b8b14a6f-e85a-4d06-adb1-6da13b14be92 2026-02-22T20:21:00.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:21:00.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:21:00.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:21:00.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:136: TEST_truncated_sna_record: sleep 1 2026-02-22T20:21:01.323 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: expr 3 - 1 2026-02-22T20:21:01.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: seq 0 2 2026-02-22T20:21:01.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-22T20:21:01.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.0 version 2026-02-22T20:21:01.393 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:21:01.394 INFO:tasks.workunit.client.0.trial150.stdout: "version": "20.2.0-794-ga5c3084f", 2026-02-22T20:21:01.394 INFO:tasks.workunit.client.0.trial150.stdout: "release": "tentacle", 2026-02-22T20:21:01.394 INFO:tasks.workunit.client.0.trial150.stdout: "release_type": "stable" 2026-02-22T20:21:01.394 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:21:01.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-22T20:21:01.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.1 version 2026-02-22T20:21:01.466 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:21:01.466 INFO:tasks.workunit.client.0.trial150.stdout: "version": "20.2.0-794-ga5c3084f", 2026-02-22T20:21:01.467 INFO:tasks.workunit.client.0.trial150.stdout: "release": "tentacle", 2026-02-22T20:21:01.467 INFO:tasks.workunit.client.0.trial150.stdout: "release_type": "stable" 2026-02-22T20:21:01.467 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:21:01.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:138: TEST_truncated_sna_record: for sdn in $(seq 0 $(expr $osdn - 1)) 2026-02-22T20:21:01.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:140: TEST_truncated_sna_record: timeout 60 ceph tell osd.2 version 2026-02-22T20:21:01.539 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:21:01.539 INFO:tasks.workunit.client.0.trial150.stdout: "version": "20.2.0-794-ga5c3084f", 2026-02-22T20:21:01.539 INFO:tasks.workunit.client.0.trial150.stdout: "release": "tentacle", 2026-02-22T20:21:01.539 INFO:tasks.workunit.client.0.trial150.stdout: "release_type": "stable" 2026-02-22T20:21:01.539 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:21:01.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:142: TEST_truncated_sna_record: rados --format json-pretty -p test listsnaps objxxx 2026-02-22T20:21:01.565 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:21:01.566 INFO:tasks.workunit.client.0.trial150.stdout: "name": "objxxx", 2026-02-22T20:21:01.566 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 5, 2026-02-22T20:21:01.566 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:21:01.566 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:21:01.566 INFO:tasks.workunit.client.0.trial150.stdout: "id": 2, 2026-02-22T20:21:01.566 INFO:tasks.workunit.client.0.trial150.stdout: "snapshots": [ 2026-02-22T20:21:01.566 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:21:01.566 INFO:tasks.workunit.client.0.trial150.stdout: "id": 1, 2026-02-22T20:21:01.566 INFO:tasks.workunit.client.0.trial150.stdout: "name": "snap01" 2026-02-22T20:21:01.566 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:21:01.567 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:21:01.567 INFO:tasks.workunit.client.0.trial150.stdout: "id": 2, 2026-02-22T20:21:01.567 INFO:tasks.workunit.client.0.trial150.stdout: "name": "snap02" 2026-02-22T20:21:01.567 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:21:01.567 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:21:01.567 INFO:tasks.workunit.client.0.trial150.stdout: "size": 6, 2026-02-22T20:21:01.567 INFO:tasks.workunit.client.0.trial150.stdout: "overlaps": [] 2026-02-22T20:21:01.567 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:21:01.567 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:21:01.567 INFO:tasks.workunit.client.0.trial150.stdout: "id": 3, 2026-02-22T20:21:01.568 INFO:tasks.workunit.client.0.trial150.stdout: "snapshots": [ 2026-02-22T20:21:01.568 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:21:01.568 INFO:tasks.workunit.client.0.trial150.stdout: "id": 3, 2026-02-22T20:21:01.568 INFO:tasks.workunit.client.0.trial150.stdout: "name": "snap13" 2026-02-22T20:21:01.568 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:21:01.568 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:21:01.568 INFO:tasks.workunit.client.0.trial150.stdout: "size": 6, 2026-02-22T20:21:01.568 INFO:tasks.workunit.client.0.trial150.stdout: "overlaps": [] 2026-02-22T20:21:01.568 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:21:01.568 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:21:01.568 INFO:tasks.workunit.client.0.trial150.stdout: "id": 5, 2026-02-22T20:21:01.568 INFO:tasks.workunit.client.0.trial150.stdout: "snapshots": [ 2026-02-22T20:21:01.569 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:21:01.569 INFO:tasks.workunit.client.0.trial150.stdout: "id": 4, 2026-02-22T20:21:01.569 INFO:tasks.workunit.client.0.trial150.stdout: "name": "snap24" 2026-02-22T20:21:01.569 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:21:01.569 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:21:01.569 INFO:tasks.workunit.client.0.trial150.stdout: "id": 5, 2026-02-22T20:21:01.569 INFO:tasks.workunit.client.0.trial150.stdout: "name": "snap25" 2026-02-22T20:21:01.569 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:21:01.569 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:21:01.569 INFO:tasks.workunit.client.0.trial150.stdout: "size": 6, 2026-02-22T20:21:01.569 INFO:tasks.workunit.client.0.trial150.stdout: "overlaps": [] 2026-02-22T20:21:01.569 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:21:01.570 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:21:01.570 INFO:tasks.workunit.client.0.trial150.stdout: "id": "head", 2026-02-22T20:21:01.570 INFO:tasks.workunit.client.0.trial150.stdout: "snapshots": [], 2026-02-22T20:21:01.570 INFO:tasks.workunit.client.0.trial150.stdout: "size": 6 2026-02-22T20:21:01.570 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:21:01.570 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:21:01.570 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:21:01.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:145: TEST_truncated_sna_record: ceph osd unset nodeep-scrub 2026-02-22T20:21:01.776 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is unset 2026-02-22T20:21:01.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:146: TEST_truncated_sna_record: ceph osd unset noscrub 2026-02-22T20:21:01.980 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is unset 2026-02-22T20:21:01.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: ceph --format=json-pretty osd map test objxxx 2026-02-22T20:21:01.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: jq -r '.up[0]' 2026-02-22T20:21:02.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:149: TEST_truncated_sna_record: local cur_prim=1 2026-02-22T20:21:02.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:150: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-22T20:21:02.300 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:21:02.300 INFO:tasks.workunit.client.0.trial150.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2026-02-22T20:21:00.259409+0000 22'7 36:66 [1,2,0] 1 [1,2,0] 1 22'7 2026-02-22T20:20:42.326834+0000 22'7 2026-02-22T20:20:42.326834+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:42.326834+0000 4 0 2026-02-22T20:21:02.300 INFO:tasks.workunit.client.0.trial150.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:21:00.259373+0000 0'0 36:40 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:21:02.300 INFO:tasks.workunit.client.0.trial150.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:21:00.259405+0000 0'0 36:47 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:21:02.300 INFO:tasks.workunit.client.0.trial150.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:21:00.259317+0000 0'0 36:34 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:21:02.300 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:21:02.300 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:21:02.300 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:21:02.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:151: TEST_truncated_sna_record: sleep 2 2026-02-22T20:21:04.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:152: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2026-02-22T20:21:04.374 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:21:04.375 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:21:04.375 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:21:04.375 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:21:04.375 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:21:04.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:153: TEST_truncated_sna_record: sleep 5 2026-02-22T20:21:09.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:154: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-22T20:21:09.532 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:21:09.532 INFO:tasks.workunit.client.0.trial150.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2026-02-22T20:21:05.293995+0000 22'7 38:80 [1,2,0] 1 [1,2,0] 1 22'7 2026-02-22T20:21:05.293922+0000 22'7 2026-02-22T20:21:05.293922+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:21:05.293922+0000 4 0 2026-02-22T20:21:09.532 INFO:tasks.workunit.client.0.trial150.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:21:00.259373+0000 0'0 38:45 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:21:09.532 INFO:tasks.workunit.client.0.trial150.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:21:00.259405+0000 0'0 38:51 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:21:09.533 INFO:tasks.workunit.client.0.trial150.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:21:00.259317+0000 0'0 38:40 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:21:09.533 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:21:09.533 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:21:09.533 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:21:09.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:21:09.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:155: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2026-02-22T20:21:09.542 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:05.290+0000 7f33bd069640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2026-02-22T20:21:09.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:156: TEST_truncated_sna_record: grep -a -q ERR td/osd-mapper/osd.1.log 2026-02-22T20:21:09.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2026-02-22T20:21:09.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: wc -l 2026-02-22T20:21:09.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:160: TEST_truncated_sna_record: local prev_err_cnt=1 2026-02-22T20:21:09.547 INFO:tasks.workunit.client.0.trial150.stdout:prev count: 1 2026-02-22T20:21:09.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:161: TEST_truncated_sna_record: echo 'prev count: 1' 2026-02-22T20:21:09.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:164: TEST_truncated_sna_record: ceph pg 1.3 deep-scrub 2026-02-22T20:21:09.614 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:21:09.614 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:21:09.614 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:21:09.614 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:21:09.614 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:21:09.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:165: TEST_truncated_sna_record: sleep 5 2026-02-22T20:21:14.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:166: TEST_truncated_sna_record: ceph pg dump pgs 2026-02-22T20:21:14.774 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:21:14.774 INFO:tasks.workunit.client.0.trial150.stdout:1.3 4 0 0 0 0 24 0 0 7 0 7 active+clean 2026-02-22T20:21:10.338450+0000 22'7 38:89 [1,2,0] 1 [1,2,0] 1 22'7 2026-02-22T20:21:10.338382+0000 22'7 2026-02-22T20:21:10.338382+0000 0 1 periodic scrub scheduled @ 2026-02-23T20:21:10.338382+0000 4 0 2026-02-22T20:21:14.775 INFO:tasks.workunit.client.0.trial150.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:21:00.259373+0000 0'0 38:46 [0,1,2] 0 [0,1,2] 0 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:21:14.775 INFO:tasks.workunit.client.0.trial150.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:21:00.259405+0000 0'0 38:52 [2,0,1] 2 [2,0,1] 2 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:21:14.775 INFO:tasks.workunit.client.0.trial150.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:21:00.259317+0000 0'0 38:41 [1,0,2] 1 [1,0,2] 1 0'0 2026-02-22T20:20:35.993255+0000 0'0 2026-02-22T20:20:35.993255+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:20:35.993255+0000 0 0 2026-02-22T20:21:14.775 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:21:14.775 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:21:14.775 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:21:14.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:21:14.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:167: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2026-02-22T20:21:14.784 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:05.290+0000 7f33bd069640 -1 log_channel(cluster) log [ERR] : osd.1 found snap mapper error on pg 1.3 oid 1:f5ae0ba1:::objxxx:3 snaps in mapper: {}, oi: {3} ...repaired 2026-02-22T20:21:14.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: grep -a ERR td/osd-mapper/osd.1.log 2026-02-22T20:21:14.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: wc -l 2026-02-22T20:21:14.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:168: TEST_truncated_sna_record: local current_err_cnt=1 2026-02-22T20:21:14.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: (( extr_dbg >= 1 )) 2026-02-22T20:21:14.788 INFO:tasks.workunit.client.0.trial150.stdout:current count: 1 2026-02-22T20:21:14.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:169: TEST_truncated_sna_record: echo 'current count: 1' 2026-02-22T20:21:14.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:170: TEST_truncated_sna_record: (( current_err_cnt == prev_err_cnt )) 2026-02-22T20:21:14.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:171: TEST_truncated_sna_record: kill_daemons td/osd-mapper TERM osd 2026-02-22T20:21:14.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:21:14.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:21:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:21:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:21:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:21:15.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:21:15.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:172: TEST_truncated_sna_record: kvdir=td/osd-mapper/1 2026-02-22T20:21:15.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: (( extr_dbg >= 2 )) 2026-02-22T20:21:15.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-22T20:21:15.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:173: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_' 2026-02-22T20:21:15.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:174: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-22T20:21:15.317 INFO:tasks.workunit.client.0.trial150.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:21:15.317 INFO:tasks.workunit.client.0.trial150.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.FA570D58.2.objxxx.. 2026-02-22T20:21:15.317 INFO:tasks.workunit.client.0.trial150.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_000000 2026-02-22T20:21:15.317 INFO:tasks.workunit.client.0.trial150.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.FA570D58.3.objxxx.. 2026-02-22T20:21:15.318 INFO:tasks.workunit.client.0.trial150.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:21:15.318 INFO:tasks.workunit.client.0.trial150.stdout:%00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.FA570D58.5.objxxx.. 2026-02-22T20:21:15.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: ceph-kvstore-tool bluestore-kv td/osd-mapper/1 dump p 2026-02-22T20:21:15.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: grep -a -e 'SNA_[0-9]_000000000000000[0-9]_000000000000000' 2026-02-22T20:21:15.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: awk -e '{print $2;}' 2026-02-22T20:21:15.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: wc -l 2026-02-22T20:21:15.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:176: TEST_truncated_sna_record: local num_sna_full=5 2026-02-22T20:21:15.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:177: TEST_truncated_sna_record: (( num_sna_full == num_sna_b4 )) 2026-02-22T20:21:15.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:178: TEST_truncated_sna_record: return 0 2026-02-22T20:21:15.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-mapper.sh:24: run: teardown td/osd-mapper 2026-02-22T20:21:15.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2026-02-22T20:21:15.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:21:15.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2026-02-22T20:21:15.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:21:15.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:21:15.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:21:15.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:21:15.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:21:15.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:21:15.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:21:15.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:21:15.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:21:15.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:21:15.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:21:15.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:21:15.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:15.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:21:15.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:21:15.680 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:15.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:21:15.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:21:15.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2026-02-22T20:21:15.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:21:15.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:15.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12039 2026-02-22T20:21:15.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12039 2026-02-22T20:21:15.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:21:15.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:21:15.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-22T20:21:15.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-mapper 0 2026-02-22T20:21:15.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-mapper 2026-02-22T20:21:15.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-22T20:21:15.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-mapper KILL 2026-02-22T20:21:15.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:21:15.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:21:15.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:21:15.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:21:15.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:21:15.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:21:15.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:21:15.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:21:15.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:21:15.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:21:15.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:21:15.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:21:15.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:15.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:21:15.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:21:15.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:15.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:21:15.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T20:21:15.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-mapper 2026-02-22T20:21:15.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:21:15.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:15.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.12039 2026-02-22T20:21:15.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.12039 2026-02-22T20:21:15.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:21:15.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:21:15.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-22T20:21:15.707 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T20:21:15.707 DEBUG:teuthology.orchestra.run.trial150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T20:21:15.758 INFO:tasks.workunit:Running workunit scrub/osd-recovery-scrub.sh... 2026-02-22T20:21:15.758 DEBUG:teuthology.orchestra.run.trial150:workunit test scrub/osd-recovery-scrub.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5c3084f688b1f6eff223b8d259a8316f9844c8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh 2026-02-22T20:21:15.806 INFO:tasks.workunit.client.0.trial150.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T20:21:15.810 INFO:tasks.workunit.client.0.trial150.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T20:21:15.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T20:21:15.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T20:21:15.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-22T20:21:15.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-22T20:21:15.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-22T20:21:15.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-22T20:21:15.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-22T20:21:15.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-22T20:21:15.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-recovery-scrub 2026-02-22T20:21:15.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:20: run: local dir=td/osd-recovery-scrub 2026-02-22T20:21:15.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:21: run: shift 2026-02-22T20:21:15.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: export CEPH_MON=127.0.0.1:7124 2026-02-22T20:21:15.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:23: run: CEPH_MON=127.0.0.1:7124 2026-02-22T20:21:15.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:24: run: export CEPH_ARGS 2026-02-22T20:21:15.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: uuidgen 2026-02-22T20:21:15.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:25: run: CEPH_ARGS+='--fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none ' 2026-02-22T20:21:15.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2026-02-22T20:21:15.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:27: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2026-02-22T20:21:15.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-22T20:21:15.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: set 2026-02-22T20:21:15.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T20:21:15.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:30: run: local funcs=TEST_recovery_scrub_1 2026-02-22T20:21:15.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:31: run: for func in $funcs 2026-02-22T20:21:15.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:32: run: TEST_recovery_scrub_1 td/osd-recovery-scrub 2026-02-22T20:21:15.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:39: TEST_recovery_scrub_1: local dir=td/osd-recovery-scrub 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:40: TEST_recovery_scrub_1: local poolname=test 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:42: TEST_recovery_scrub_1: TESTDATA=testdata.31002 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:43: TEST_recovery_scrub_1: OSDS=4 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:44: TEST_recovery_scrub_1: PGS=1 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:45: TEST_recovery_scrub_1: OBJECTS=100 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:46: TEST_recovery_scrub_1: ERRORS=0 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:48: TEST_recovery_scrub_1: setup td/osd-recovery-scrub 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-recovery-scrub 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-recovery-scrub 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2026-02-22T20:21:15.816 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:21:15.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:21:15.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:21:15.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:21:15.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:21:15.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:21:15.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:21:15.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:21:15.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:21:15.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:21:15.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:21:15.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:21:15.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:15.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:21:15.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:21:15.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:15.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:21:15.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:21:15.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2026-02-22T20:21:15.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:21:15.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:15.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.31002 2026-02-22T20:21:15.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.31002 2026-02-22T20:21:15.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:21:15.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:21:15.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-recovery-scrub 2026-02-22T20:21:15.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:21:15.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:15.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.31002 2026-02-22T20:21:15.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.31002 2026-02-22T20:21:15.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:21:15.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-22T20:21:15.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-22T20:21:15.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:21:15.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-recovery-scrub 1' TERM HUP INT 2026-02-22T20:21:15.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:49: TEST_recovery_scrub_1: run_mon td/osd-recovery-scrub a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T20:21:15.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-recovery-scrub 2026-02-22T20:21:15.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:21:15.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:21:15.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:21:15.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-recovery-scrub/a 2026-02-22T20:21:15.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-scrub/a --run-dir=td/osd-recovery-scrub --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T20:21:15.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:21:15.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:21:15.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:21:15.854 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:21:15.854 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:15.854 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.31002 2026-02-22T20:21:15.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.31002/$cluster-$name.asok' 2026-02-22T20:21:15.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-scrub/a '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-scrub/log --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T20:21:15.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:21:15.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:21:15.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:21:15.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:21:15.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:21:15.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:21:15.881 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:21:15.882 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:21:15.882 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:21:15.883 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:21:15.883 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:15.883 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.31002 2026-02-22T20:21:15.883 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.31002/ceph-mon.a.asok 2026-02-22T20:21:15.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:21:15.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.31002/ceph-mon.a.asok config get fsid 2026-02-22T20:21:15.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:21:15.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:21:15.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:21:15.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:21:15.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:21:15.937 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:21:15.937 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:21:15.937 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:21:15.937 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:21:15.937 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:15.937 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.31002 2026-02-22T20:21:15.937 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.31002/ceph-mon.a.asok 2026-02-22T20:21:15.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:21:15.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.31002/ceph-mon.a.asok config get mon_host 2026-02-22T20:21:15.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:50: TEST_recovery_scrub_1: run_mgr td/osd-recovery-scrub x 2026-02-22T20:21:15.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-recovery-scrub 2026-02-22T20:21:15.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:21:15.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:21:15.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:21:15.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-recovery-scrub/x 2026-02-22T20:21:15.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:21:16.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:21:16.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:21:16.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:21:16.085 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:21:16.085 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:16.085 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.31002 2026-02-22T20:21:16.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.31002/$cluster-$name.asok' 2026-02-22T20:21:16.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:21:16.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-scrub/x '--log-file=td/osd-recovery-scrub/$name.log' '--admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' --run-dir=td/osd-recovery-scrub '--pid-file=td/osd-recovery-scrub/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:21:16.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:51: TEST_recovery_scrub_1: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 ' 2026-02-22T20:21:16.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:52: TEST_recovery_scrub_1: ceph_osd_args+='--osd_scrub_backoff_ratio=0 ' 2026-02-22T20:21:16.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:53: TEST_recovery_scrub_1: ceph_osd_args+='--osd_stats_update_period_not_scrubbing=3 ' 2026-02-22T20:21:16.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:54: TEST_recovery_scrub_1: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2026-02-22T20:21:16.103 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: expr 4 - 1 2026-02-22T20:21:16.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: seq 0 3 2026-02-22T20:21:16.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:21:16.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 0 --osd_scrub_during_recovery=false 2026-02-22T20:21:16.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2026-02-22T20:21:16.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:21:16.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:21:16.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:21:16.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/0 2026-02-22T20:21:16.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2026-02-22T20:21:16.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:21:16.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:21:16.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:21:16.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/0' 2026-02-22T20:21:16.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/0/journal' 2026-02-22T20:21:16.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:21:16.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:21:16.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2026-02-22T20:21:16.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:21:16.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:21:16.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:21:16.108 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:21:16.108 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:16.108 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.31002 2026-02-22T20:21:16.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.31002/$cluster-$name.asok' 2026-02-22T20:21:16.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' 2026-02-22T20:21:16.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:21:16.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:21:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:21:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2026-02-22T20:21:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2026-02-22T20:21:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:21:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:21:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:21:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:21:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:21:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2026-02-22T20:21:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/0 2026-02-22T20:21:16.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:21:16.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=10d427c9-de0a-4709-8569-6c9d6c91c8cf 2026-02-22T20:21:16.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 10d427c9-de0a-4709-8569-6c9d6c91c8cf' 2026-02-22T20:21:16.111 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 10d427c9-de0a-4709-8569-6c9d6c91c8cf 2026-02-22T20:21:16.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:21:16.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA8ZZtpCHDmBhAA4e+cfLRaRuo2sD162WeluQ== 2026-02-22T20:21:16.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA8ZZtpCHDmBhAA4e+cfLRaRuo2sD162WeluQ=="}' 2026-02-22T20:21:16.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 10d427c9-de0a-4709-8569-6c9d6c91c8cf -i td/osd-recovery-scrub/0/new.json 2026-02-22T20:21:16.209 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:21:16.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/0/new.json 2026-02-22T20:21:16.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQA8ZZtpCHDmBhAA4e+cfLRaRuo2sD162WeluQ== --osd-uuid 10d427c9-de0a-4709-8569-6c9d6c91c8cf 2026-02-22T20:21:16.233 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:16.226+0000 7fbbe71d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:16.235 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:16.230+0000 7fbbe71d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:16.236 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:16.230+0000 7fbbe71d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:16.236 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:16.230+0000 7fbbe71d8a40 -1 bdev(0x56175a768c00 td/osd-recovery-scrub/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:21:16.236 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:16.230+0000 7fbbe71d8a40 -1 bluestore(td/osd-recovery-scrub/0) _read_fsid unparsable uuid 2026-02-22T20:21:16.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/0/keyring 2026-02-22T20:21:16.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:21:16.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:21:16.939 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:21:16.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:21:17.151 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:21:17.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:21:17.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/0 --osd-journal=td/osd-recovery-scrub/0/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2026-02-22T20:21:17.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:21:17.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:21:17.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:21:17.175 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:17.170+0000 7f9b2a4e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:17.177 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:17.170+0000 7f9b2a4e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:17.178 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:17.170+0000 7f9b2a4e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:17.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:21:17.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:21:17.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:21:17.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:21:17.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:21:17.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:21:17.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:21:17.308 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:21:17.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:21:17.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:21:17.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:21:17.596 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:17.590+0000 7f9b2a4e3a40 -1 Falling back to public interface 2026-02-22T20:21:18.200 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:18.194+0000 7f9b2a4e3a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:21:18.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:21:18.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:21:18.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:21:18.466 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:21:18.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:21:18.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:21:18.625 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1547813204,v1:127.0.0.1:6803/1547813204] [v2:127.0.0.1:6804/1547813204,v1:127.0.0.1:6805/1547813204] exists,up 10d427c9-de0a-4709-8569-6c9d6c91c8cf 2026-02-22T20:21:18.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 1 --osd_scrub_during_recovery=false 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/1 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/1' 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/1/journal' 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:21:18.626 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.31002 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.31002/$cluster-$name.asok' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/1 2026-02-22T20:21:18.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:21:18.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2da2bcc1-9978-46b3-a36b-ef3a031a5830 2026-02-22T20:21:18.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2da2bcc1-9978-46b3-a36b-ef3a031a5830' 2026-02-22T20:21:18.628 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 2da2bcc1-9978-46b3-a36b-ef3a031a5830 2026-02-22T20:21:18.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:21:18.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA+ZZtpux4DJhAA5d7ssjlmhgymj71r9xmKdw== 2026-02-22T20:21:18.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA+ZZtpux4DJhAA5d7ssjlmhgymj71r9xmKdw=="}' 2026-02-22T20:21:18.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2da2bcc1-9978-46b3-a36b-ef3a031a5830 -i td/osd-recovery-scrub/1/new.json 2026-02-22T20:21:18.807 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:21:18.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/1/new.json 2026-02-22T20:21:18.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQA+ZZtpux4DJhAA5d7ssjlmhgymj71r9xmKdw== --osd-uuid 2da2bcc1-9978-46b3-a36b-ef3a031a5830 2026-02-22T20:21:18.833 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:18.826+0000 7f2892ea1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:18.834 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:18.830+0000 7f2892ea1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:18.835 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:18.830+0000 7f2892ea1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:18.836 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:18.830+0000 7f2892ea1a40 -1 bdev(0x555d32870c00 td/osd-recovery-scrub/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:21:18.836 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:18.830+0000 7f2892ea1a40 -1 bluestore(td/osd-recovery-scrub/1) _read_fsid unparsable uuid 2026-02-22T20:21:19.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/1/keyring 2026-02-22T20:21:19.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:21:19.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:21:19.462 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:21:19.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:21:19.661 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:21:19.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:21:19.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/1 --osd-journal=td/osd-recovery-scrub/1/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2026-02-22T20:21:19.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:21:19.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:21:19.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:21:19.686 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:19.682+0000 7f626e78fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:19.688 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:19.682+0000 7f626e78fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:19.689 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:19.682+0000 7f626e78fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:19.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:21:19.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:21:19.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:21:19.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:21:19.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:21:19.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:21:19.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:21:19.819 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:21:19.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:21:19.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:21:19.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:21:20.044 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:20.038+0000 7f626e78fa40 -1 Falling back to public interface 2026-02-22T20:21:20.586 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:20.578+0000 7f626e78fa40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:21:20.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:21:20.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:21:20.976 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:21:20.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:21:20.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:21:20.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:21:21.136 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2220991320,v1:127.0.0.1:6811/2220991320] [v2:127.0.0.1:6812/2220991320,v1:127.0.0.1:6813/2220991320] exists,up 2da2bcc1-9978-46b3-a36b-ef3a031a5830 2026-02-22T20:21:21.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:21:21.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:21:21.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:21:21.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:21:21.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 2 --osd_scrub_during_recovery=false 2026-02-22T20:21:21.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2026-02-22T20:21:21.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:21:21.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:21:21.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:21:21.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/2 2026-02-22T20:21:21.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2026-02-22T20:21:21.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:21:21.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:21:21.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:21:21.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/2' 2026-02-22T20:21:21.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/2/journal' 2026-02-22T20:21:21.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:21:21.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:21:21.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2026-02-22T20:21:21.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:21:21.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:21:21.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:21:21.143 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:21:21.143 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:21.143 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.31002 2026-02-22T20:21:21.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.31002/$cluster-$name.asok' 2026-02-22T20:21:21.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' 2026-02-22T20:21:21.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:21:21.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:21:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:21:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2026-02-22T20:21:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2026-02-22T20:21:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:21:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:21:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:21:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:21:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:21:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2026-02-22T20:21:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/2 2026-02-22T20:21:21.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:21:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=878ca4c6-7809-4734-97b2-2f05e1df1a83 2026-02-22T20:21:21.145 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 878ca4c6-7809-4734-97b2-2f05e1df1a83 2026-02-22T20:21:21.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 878ca4c6-7809-4734-97b2-2f05e1df1a83' 2026-02-22T20:21:21.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:21:21.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBZZtpZZ7iCBAARvqJcUomAqpQE266TAdSJg== 2026-02-22T20:21:21.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBZZtpZZ7iCBAARvqJcUomAqpQE266TAdSJg=="}' 2026-02-22T20:21:21.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 878ca4c6-7809-4734-97b2-2f05e1df1a83 -i td/osd-recovery-scrub/2/new.json 2026-02-22T20:21:21.306 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:21:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/2/new.json 2026-02-22T20:21:21.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBBZZtpZZ7iCBAARvqJcUomAqpQE266TAdSJg== --osd-uuid 878ca4c6-7809-4734-97b2-2f05e1df1a83 2026-02-22T20:21:21.332 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:21.326+0000 7f332692ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:21.334 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:21.326+0000 7f332692ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:21.335 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:21.330+0000 7f332692ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:21.335 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:21.330+0000 7f332692ba40 -1 bdev(0x5613cd3f8c00 td/osd-recovery-scrub/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:21:21.335 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:21.330+0000 7f332692ba40 -1 bluestore(td/osd-recovery-scrub/2) _read_fsid unparsable uuid 2026-02-22T20:21:21.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/2/keyring 2026-02-22T20:21:21.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:21:21.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:21:21.937 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:21:21.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:21:22.142 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:21:22.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:21:22.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/2 --osd-journal=td/osd-recovery-scrub/2/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2026-02-22T20:21:22.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:21:22.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:21:22.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:21:22.165 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:22.158+0000 7f65ea228a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:22.167 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:22.162+0000 7f65ea228a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:22.168 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:22.162+0000 7f65ea228a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:22.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:21:22.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:21:22.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:21:22.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:21:22.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:21:22.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:21:22.304 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:21:22.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:21:22.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:21:22.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:21:22.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:21:22.540 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:22.534+0000 7f65ea228a40 -1 Falling back to public interface 2026-02-22T20:21:23.193 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:23.186+0000 7f65ea228a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:21:23.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:21:23.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:21:23.462 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:21:23.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:21:23.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:21:23.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:21:23.618 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3150337278,v1:127.0.0.1:6819/3150337278] [v2:127.0.0.1:6820/3150337278,v1:127.0.0.1:6821/3150337278] exists,up 878ca4c6-7809-4734-97b2-2f05e1df1a83 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:55: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:57: TEST_recovery_scrub_1: run_osd td/osd-recovery-scrub 3 --osd_scrub_during_recovery=false 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-recovery-scrub 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-recovery-scrub/3 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq ' 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-recovery-scrub/3' 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-scrub/3/journal' 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:21:23.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-recovery-scrub' 2026-02-22T20:21:23.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:21:23.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:21:23.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:21:23.620 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:21:23.620 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:23.620 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.31002 2026-02-22T20:21:23.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.31002/$cluster-$name.asok' 2026-02-22T20:21:23.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' 2026-02-22T20:21:23.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:21:23.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:21:23.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:21:23.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-recovery-scrub/$name.log' 2026-02-22T20:21:23.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-recovery-scrub/$name.pid' 2026-02-22T20:21:23.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:21:23.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:21:23.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:21:23.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:21:23.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:21:23.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_during_recovery=false 2026-02-22T20:21:23.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-recovery-scrub/3 2026-02-22T20:21:23.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:21:23.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=43b88867-9359-4f07-9385-eba538bf404d 2026-02-22T20:21:23.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 43b88867-9359-4f07-9385-eba538bf404d' 2026-02-22T20:21:23.623 INFO:tasks.workunit.client.0.trial150.stdout:add osd3 43b88867-9359-4f07-9385-eba538bf404d 2026-02-22T20:21:23.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:21:23.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBDZZtpDaG5JRAAA6l0eNsOxqXYf3Hq5FU8dw== 2026-02-22T20:21:23.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBDZZtpDaG5JRAAA6l0eNsOxqXYf3Hq5FU8dw=="}' 2026-02-22T20:21:23.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 43b88867-9359-4f07-9385-eba538bf404d -i td/osd-recovery-scrub/3/new.json 2026-02-22T20:21:23.797 INFO:tasks.workunit.client.0.trial150.stdout:3 2026-02-22T20:21:23.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-recovery-scrub/3/new.json 2026-02-22T20:21:23.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --mkfs --key AQBDZZtpDaG5JRAAA6l0eNsOxqXYf3Hq5FU8dw== --osd-uuid 43b88867-9359-4f07-9385-eba538bf404d 2026-02-22T20:21:23.824 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:23.818+0000 7f18b645ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:23.826 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:23.818+0000 7f18b645ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:23.827 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:23.822+0000 7f18b645ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:23.827 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:23.822+0000 7f18b645ca40 -1 bdev(0x56488ddf4c00 td/osd-recovery-scrub/3/block) open stat got: (1) Operation not permitted 2026-02-22T20:21:23.828 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:23.822+0000 7f18b645ca40 -1 bluestore(td/osd-recovery-scrub/3) _read_fsid unparsable uuid 2026-02-22T20:21:24.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-recovery-scrub/3/keyring 2026-02-22T20:21:24.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:21:24.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-22T20:21:24.446 INFO:tasks.workunit.client.0.trial150.stdout:adding osd3 key to auth repository 2026-02-22T20:21:24.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-recovery-scrub/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:21:24.650 INFO:tasks.workunit.client.0.trial150.stdout:start osd.3 2026-02-22T20:21:24.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-22T20:21:24.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=8cd78f6c-0252-469c-b4fc-e44a933b6c6a --auth-supported=none --mon-host=127.0.0.1:7124 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-scrub/3 --osd-journal=td/osd-recovery-scrub/3/journal --chdir= --run-dir=td/osd-recovery-scrub '--admin-socket=/tmp/ceph-asok.31002/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-scrub/$name.log' '--pid-file=td/osd-recovery-scrub/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false 2026-02-22T20:21:24.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:21:24.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:21:24.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:21:24.675 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:24.670+0000 7f3931bcea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:24.677 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:24.670+0000 7f3931bcea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:24.678 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:24.674+0000 7f3931bcea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:24.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-22T20:21:24.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:21:24.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:21:24.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:21:24.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:21:24.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:21:24.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:21:24.808 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:21:24.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:21:24.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:21:24.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:21:25.052 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:25.046+0000 7f3931bcea40 -1 Falling back to public interface 2026-02-22T20:21:25.729 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:25.722+0000 7f3931bcea40 -1 osd.3 0 log_to_monitors true 2026-02-22T20:21:25.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:21:25.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:21:25.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:21:25.968 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:21:25.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:21:25.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:21:26.123 INFO:tasks.workunit.client.0.trial150.stdout:osd.3 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3351260842,v1:127.0.0.1:6827/3351260842] [v2:127.0.0.1:6828/3351260842,v1:127.0.0.1:6829/3351260842] exists,up 43b88867-9359-4f07-9385-eba538bf404d 2026-02-22T20:21:26.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:21:26.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:21:26.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:21:26.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:61: TEST_recovery_scrub_1: create_pool test 1 1 2026-02-22T20:21:26.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-22T20:21:26.290 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T20:21:26.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:21:27.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:62: TEST_recovery_scrub_1: wait_for_clean 2026-02-22T20:21:27.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:21:27.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:21:27.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:21:27.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:21:27.304 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:21:27.304 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:21:27.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:21:27.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:21:27.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:21:27.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:21:27.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:21:27.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:21:27.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:21:27.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:21:27.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:21:27.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:21:27.571 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:21:27.571 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:21:27.571 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:21:27.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:21:27.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:21:27.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:21:27.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:21:27.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:21:27.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:21:27.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:21:27.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:21:27.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-22T20:21:27.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-22T20:21:27.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-22T20:21:27.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:21:27.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:21:27.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574850 2026-02-22T20:21:27.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574850 2026-02-22T20:21:27.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667 2-55834574850' 2026-02-22T20:21:27.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:21:27.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:21:27.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444034 2026-02-22T20:21:27.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444034 2026-02-22T20:21:27.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667 2-55834574850 3-73014444034' 2026-02-22T20:21:27.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:21:27.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:21:27.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:21:27.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:21:27.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:21:27.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:21:27.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:21:27.861 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:21:27.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:21:27.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:21:28.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:21:28.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:21:29.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:21:29.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:21:29.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T20:21:29.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:21:29.180 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-22T20:21:29.180 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:21:29.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:21:29.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-22T20:21:29.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:21:29.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-22T20:21:29.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-22T20:21:29.183 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705667 2026-02-22T20:21:29.184 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:21:29.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-22T20:21:29.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:21:29.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574850 2026-02-22T20:21:29.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:21:29.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:21:29.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574850 2026-02-22T20:21:29.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:21:29.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574850 2026-02-22T20:21:29.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2026-02-22T20:21:29.342 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574850 2026-02-22T20:21:29.342 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:21:29.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574850 -lt 55834574850 2026-02-22T20:21:29.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:21:29.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-73014444034 2026-02-22T20:21:29.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:21:29.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:21:29.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-73014444034 2026-02-22T20:21:29.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:21:29.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=73014444034 2026-02-22T20:21:29.519 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.3 seq 73014444034 2026-02-22T20:21:29.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 73014444034' 2026-02-22T20:21:29.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:21:29.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 73014444034 -lt 73014444034 2026-02-22T20:21:29.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:21:29.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:21:29.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:21:29.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:21:29.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:21:29.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:21:29.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:21:29.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:21:29.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:21:29.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:21:29.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:21:30.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:21:30.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:21:30.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:21:30.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:21:30.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:21:30.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:21:30.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:21:30.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: ceph osd dump 2026-02-22T20:21:30.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: grep '^pool.*['\'']test['\'']' 2026-02-22T20:21:30.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: awk '{ print $2 }' 2026-02-22T20:21:30.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:63: TEST_recovery_scrub_1: poolid=1 2026-02-22T20:21:30.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:65: TEST_recovery_scrub_1: ceph pg dump pgs 2026-02-22T20:21:30.525 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:21:30.526 INFO:tasks.workunit.client.0.trial150.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2026-02-22T20:21:26.599566+0000 0'0 19:9 [1] 1 [1] 1 0'0 2026-02-22T20:21:26.285204+0000 0'0 2026-02-22T20:21:26.285204+0000 0 0 periodic scrub scheduled @ 2026-02-24T04:56:56.853740+0000 0 0 2026-02-22T20:21:30.526 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:21:30.526 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:21:30.526 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:21:30.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:67: TEST_recovery_scrub_1: dd if=/dev/urandom of=testdata.31002 bs=1M count=50 2026-02-22T20:21:30.662 INFO:tasks.workunit.client.0.trial150.stderr:50+0 records in 2026-02-22T20:21:30.662 INFO:tasks.workunit.client.0.trial150.stderr:50+0 records out 2026-02-22T20:21:30.662 INFO:tasks.workunit.client.0.trial150.stderr:52428800 bytes (52 MB, 50 MiB) copied, 0.124508 s, 421 MB/s 2026-02-22T20:21:30.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: seq 1 100 2026-02-22T20:21:30.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:30.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj1 testdata.31002 2026-02-22T20:21:30.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:30.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj2 testdata.31002 2026-02-22T20:21:30.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:30.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj3 testdata.31002 2026-02-22T20:21:30.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:30.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj4 testdata.31002 2026-02-22T20:21:30.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:30.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj5 testdata.31002 2026-02-22T20:21:31.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:31.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj6 testdata.31002 2026-02-22T20:21:31.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:31.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj7 testdata.31002 2026-02-22T20:21:31.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:31.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj8 testdata.31002 2026-02-22T20:21:31.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:31.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj9 testdata.31002 2026-02-22T20:21:31.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:31.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj10 testdata.31002 2026-02-22T20:21:31.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:31.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj11 testdata.31002 2026-02-22T20:21:31.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:31.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj12 testdata.31002 2026-02-22T20:21:31.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:31.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj13 testdata.31002 2026-02-22T20:21:31.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:31.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj14 testdata.31002 2026-02-22T20:21:31.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:31.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj15 testdata.31002 2026-02-22T20:21:31.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:31.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj16 testdata.31002 2026-02-22T20:21:31.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:31.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj17 testdata.31002 2026-02-22T20:21:32.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj18 testdata.31002 2026-02-22T20:21:32.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj19 testdata.31002 2026-02-22T20:21:32.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj20 testdata.31002 2026-02-22T20:21:32.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj21 testdata.31002 2026-02-22T20:21:32.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj22 testdata.31002 2026-02-22T20:21:32.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj23 testdata.31002 2026-02-22T20:21:32.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj24 testdata.31002 2026-02-22T20:21:32.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj25 testdata.31002 2026-02-22T20:21:32.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj26 testdata.31002 2026-02-22T20:21:32.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj27 testdata.31002 2026-02-22T20:21:32.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj28 testdata.31002 2026-02-22T20:21:32.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj29 testdata.31002 2026-02-22T20:21:32.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:32.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj30 testdata.31002 2026-02-22T20:21:33.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj31 testdata.31002 2026-02-22T20:21:33.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj32 testdata.31002 2026-02-22T20:21:33.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj33 testdata.31002 2026-02-22T20:21:33.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj34 testdata.31002 2026-02-22T20:21:33.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj35 testdata.31002 2026-02-22T20:21:33.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj36 testdata.31002 2026-02-22T20:21:33.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj37 testdata.31002 2026-02-22T20:21:33.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj38 testdata.31002 2026-02-22T20:21:33.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj39 testdata.31002 2026-02-22T20:21:33.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj40 testdata.31002 2026-02-22T20:21:33.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj41 testdata.31002 2026-02-22T20:21:33.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj42 testdata.31002 2026-02-22T20:21:33.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:33.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj43 testdata.31002 2026-02-22T20:21:34.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:34.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj44 testdata.31002 2026-02-22T20:21:34.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:34.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj45 testdata.31002 2026-02-22T20:21:34.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:34.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj46 testdata.31002 2026-02-22T20:21:34.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:34.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj47 testdata.31002 2026-02-22T20:21:34.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:34.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj48 testdata.31002 2026-02-22T20:21:34.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:34.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj49 testdata.31002 2026-02-22T20:21:34.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:34.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj50 testdata.31002 2026-02-22T20:21:34.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:34.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj51 testdata.31002 2026-02-22T20:21:34.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:34.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj52 testdata.31002 2026-02-22T20:21:34.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:34.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj53 testdata.31002 2026-02-22T20:21:34.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:34.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj54 testdata.31002 2026-02-22T20:21:35.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:35.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj55 testdata.31002 2026-02-22T20:21:35.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:35.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj56 testdata.31002 2026-02-22T20:21:35.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:35.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj57 testdata.31002 2026-02-22T20:21:35.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:35.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj58 testdata.31002 2026-02-22T20:21:35.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:35.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj59 testdata.31002 2026-02-22T20:21:35.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:35.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj60 testdata.31002 2026-02-22T20:21:35.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:35.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj61 testdata.31002 2026-02-22T20:21:35.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:35.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj62 testdata.31002 2026-02-22T20:21:35.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:35.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj63 testdata.31002 2026-02-22T20:21:35.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:35.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj64 testdata.31002 2026-02-22T20:21:35.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:35.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj65 testdata.31002 2026-02-22T20:21:35.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:35.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj66 testdata.31002 2026-02-22T20:21:36.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:36.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj67 testdata.31002 2026-02-22T20:21:36.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:36.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj68 testdata.31002 2026-02-22T20:21:36.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:36.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj69 testdata.31002 2026-02-22T20:21:36.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:36.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj70 testdata.31002 2026-02-22T20:21:36.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:36.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj71 testdata.31002 2026-02-22T20:21:36.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:36.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj72 testdata.31002 2026-02-22T20:21:36.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:36.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj73 testdata.31002 2026-02-22T20:21:36.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:36.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj74 testdata.31002 2026-02-22T20:21:36.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:36.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj75 testdata.31002 2026-02-22T20:21:36.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:36.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj76 testdata.31002 2026-02-22T20:21:36.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:36.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj77 testdata.31002 2026-02-22T20:21:36.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:36.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj78 testdata.31002 2026-02-22T20:21:37.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:37.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj79 testdata.31002 2026-02-22T20:21:37.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:37.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj80 testdata.31002 2026-02-22T20:21:37.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:37.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj81 testdata.31002 2026-02-22T20:21:37.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:37.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj82 testdata.31002 2026-02-22T20:21:37.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:37.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj83 testdata.31002 2026-02-22T20:21:37.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:37.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj84 testdata.31002 2026-02-22T20:21:37.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:37.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj85 testdata.31002 2026-02-22T20:21:37.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:37.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj86 testdata.31002 2026-02-22T20:21:37.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:37.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj87 testdata.31002 2026-02-22T20:21:37.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:37.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj88 testdata.31002 2026-02-22T20:21:37.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:37.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj89 testdata.31002 2026-02-22T20:21:37.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:37.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj90 testdata.31002 2026-02-22T20:21:38.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:38.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj91 testdata.31002 2026-02-22T20:21:38.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:38.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj92 testdata.31002 2026-02-22T20:21:38.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:38.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj93 testdata.31002 2026-02-22T20:21:38.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:38.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj94 testdata.31002 2026-02-22T20:21:38.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:38.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj95 testdata.31002 2026-02-22T20:21:38.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:38.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj96 testdata.31002 2026-02-22T20:21:38.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:38.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj97 testdata.31002 2026-02-22T20:21:38.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:38.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj98 testdata.31002 2026-02-22T20:21:38.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:38.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj99 testdata.31002 2026-02-22T20:21:38.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:68: TEST_recovery_scrub_1: for i in $(seq 1 $OBJECTS) 2026-02-22T20:21:38.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:70: TEST_recovery_scrub_1: rados -p test put obj100 testdata.31002 2026-02-22T20:21:38.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:72: TEST_recovery_scrub_1: rm -f testdata.31002 2026-02-22T20:21:38.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:74: TEST_recovery_scrub_1: ceph osd pool set test size 4 2026-02-22T20:21:39.154 INFO:tasks.workunit.client.0.trial150.stderr:set pool 1 size to 4 2026-02-22T20:21:39.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:77: TEST_recovery_scrub_1: set -o pipefail 2026-02-22T20:21:39.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:78: TEST_recovery_scrub_1: count=0 2026-02-22T20:21:39.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2026-02-22T20:21:39.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2026-02-22T20:21:39.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2026-02-22T20:21:39.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2026-02-22T20:21:39.316 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:21:39.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2026-02-22T20:21:41.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 0 -eq 10 2026-02-22T20:21:41.327 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 0 + 1 2026-02-22T20:21:41.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=1 2026-02-22T20:21:41.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2026-02-22T20:21:41.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2026-02-22T20:21:41.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2026-02-22T20:21:41.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2026-02-22T20:21:41.483 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:21:41.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:86: TEST_recovery_scrub_1: sleep 2 2026-02-22T20:21:43.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:87: TEST_recovery_scrub_1: test 1 -eq 10 2026-02-22T20:21:43.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: expr 1 + 1 2026-02-22T20:21:43.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:92: TEST_recovery_scrub_1: count=2 2026-02-22T20:21:43.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:79: TEST_recovery_scrub_1: true 2026-02-22T20:21:43.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:81: TEST_recovery_scrub_1: ceph --format json pg dump pgs 2026-02-22T20:21:43.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: jq '.pg_stats | [.[] | .state | contains("recovering")]' 2026-02-22T20:21:43.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:82: TEST_recovery_scrub_1: grep -q true 2026-02-22T20:21:43.654 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:21:43.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:84: TEST_recovery_scrub_1: break 2026-02-22T20:21:43.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:94: TEST_recovery_scrub_1: set +o pipefail 2026-02-22T20:21:43.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:95: TEST_recovery_scrub_1: ceph pg dump pgs 2026-02-22T20:21:43.817 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:21:43.817 INFO:tasks.workunit.client.0.trial150.stdout:1.0 100 0 297 0 0 5242880000 0 0 1300 0 1300 active+recovering+undersized+degraded+remapped 2026-02-22T20:21:41.413365+0000 21'1300 26:1329 [1,0,2,3] 1 [1,0] 1 0'0 2026-02-22T20:21:26.285204+0000 0'0 2026-02-22T20:21:26.285204+0000 0 0 no scrub is scheduled 0 0 2026-02-22T20:21:43.817 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:21:43.817 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:21:43.817 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:21:43.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:97: TEST_recovery_scrub_1: sleep 10 2026-02-22T20:21:53.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:99: TEST_recovery_scrub_1: kill_daemons td/osd-recovery-scrub 2026-02-22T20:21:53.830 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:21:53.830 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:21:53.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:21:53.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:21:53.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:21:59.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:21:59.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:101: TEST_recovery_scrub_1: declare -a err_strings 2026-02-22T20:21:59.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:102: TEST_recovery_scrub_1: err_strings[0]='recovery in progress.*scrubs' 2026-02-22T20:21:59.147 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: expr 4 - 1 2026-02-22T20:21:59.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: seq 0 3 2026-02-22T20:21:59.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:21:59.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2026-02-22T20:21:59.153 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:42.261+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.154 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:43.237+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.154 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:44.257+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.154 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:45.213+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.154 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:46.213+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.154 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:47.245+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.154 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:48.253+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.154 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:49.301+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.154 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:50.309+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.154 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:51.353+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.154 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:52.333+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.155 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:53.329+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.155 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:54.285+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.155 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:55.249+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.155 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:56.209+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.155 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:57.197+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.155 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:58.165+0000 7f9b22c96640 15 osd.0 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:21:59.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2026-02-22T20:21:59.168 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:41.461+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.169 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:42.421+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.169 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:43.377+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.169 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:44.381+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.169 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:45.333+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.169 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:46.317+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.169 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:47.369+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.169 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:48.365+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.169 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:49.397+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.169 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:50.445+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.169 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:51.405+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.169 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:52.449+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.170 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:53.461+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.170 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:54.469+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.170 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:55.477+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.170 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:56.481+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.170 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:57.485+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.170 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:58.477+0000 7f6266f42640 15 osd.1 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:21:59.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2026-02-22T20:21:59.172 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:42.289+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.172 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:43.249+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.172 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:44.249+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.172 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:45.293+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.172 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:46.289+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.172 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:47.273+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.172 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:48.249+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.172 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:49.205+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.172 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:50.245+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.172 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:51.257+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.173 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:52.269+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.173 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:53.309+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.173 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:54.333+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.173 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:55.381+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.173 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:56.401+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.173 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:57.441+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.173 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:58.477+0000 7f65e29db640 15 osd.2 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:104: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:21:59.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:106: TEST_recovery_scrub_1: grep 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2026-02-22T20:21:59.177 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:41.589+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.177 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:42.589+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.177 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:43.601+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.177 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:44.589+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.177 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:45.549+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.177 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:46.593+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.177 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:47.589+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.177 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:48.577+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.178 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:49.581+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.178 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:50.581+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.178 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:51.553+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.178 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:52.565+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.178 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:53.549+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.178 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:54.585+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.178 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:55.629+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.178 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:56.597+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.178 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:57.589+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.178 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:21:58.605+0000 7f392a381640 15 osd.3 osd-scrub:restrictions_on_scrubbing: recovery in progress. Operator-initiated scrubs only. 2026-02-22T20:21:59.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:108: TEST_recovery_scrub_1: for err_string in "${err_strings[@]}" 2026-02-22T20:21:59.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:110: TEST_recovery_scrub_1: found=false 2026-02-22T20:21:59.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:111: TEST_recovery_scrub_1: count=0 2026-02-22T20:21:59.179 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: expr 4 - 1 2026-02-22T20:21:59.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: seq 0 3 2026-02-22T20:21:59.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:21:59.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.0.log 2026-02-22T20:21:59.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2026-02-22T20:21:59.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 0 + 1 2026-02-22T20:21:59.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=1 2026-02-22T20:21:59.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:21:59.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.1.log 2026-02-22T20:21:59.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2026-02-22T20:21:59.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 1 + 1 2026-02-22T20:21:59.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=2 2026-02-22T20:21:59.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:21:59.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.2.log 2026-02-22T20:21:59.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2026-02-22T20:21:59.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 2 + 1 2026-02-22T20:21:59.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=3 2026-02-22T20:21:59.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:112: TEST_recovery_scrub_1: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:21:59.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:114: TEST_recovery_scrub_1: grep -q 'recovery in progress.*scrubs' td/osd-recovery-scrub/osd.3.log 2026-02-22T20:21:59.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:116: TEST_recovery_scrub_1: found=true 2026-02-22T20:21:59.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: expr 3 + 1 2026-02-22T20:21:59.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:117: TEST_recovery_scrub_1: count=4 2026-02-22T20:21:59.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:120: TEST_recovery_scrub_1: '[' true = false ']' 2026-02-22T20:21:59.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:124: TEST_recovery_scrub_1: '[' 4 -eq 4 ']' 2026-02-22T20:21:59.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:127: TEST_recovery_scrub_1: teardown td/osd-recovery-scrub 2026-02-22T20:21:59.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2026-02-22T20:21:59.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:21:59.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2026-02-22T20:21:59.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:21:59.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:21:59.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:21:59.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:21:59.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:21:59.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:21:59.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:21:59.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:21:59.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:21:59.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:21:59.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:21:59.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:21:59.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:59.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:21:59.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:21:59.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:59.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:21:59.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:21:59.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2026-02-22T20:21:59.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:21:59.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:59.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.31002 2026-02-22T20:21:59.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.31002 2026-02-22T20:21:59.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:21:59.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:21:59.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:129: TEST_recovery_scrub_1: '[' 0 '!=' 0 ']' 2026-02-22T20:21:59.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:135: TEST_recovery_scrub_1: echo 'TEST PASSED' 2026-02-22T20:21:59.257 INFO:tasks.workunit.client.0.trial150.stdout:TEST PASSED 2026-02-22T20:21:59.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-recovery-scrub.sh:136: TEST_recovery_scrub_1: return 0 2026-02-22T20:21:59.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-22T20:21:59.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-recovery-scrub 0 2026-02-22T20:21:59.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-recovery-scrub 2026-02-22T20:21:59.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-22T20:21:59.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-recovery-scrub KILL 2026-02-22T20:21:59.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:21:59.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:21:59.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:21:59.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:21:59.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:21:59.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:21:59.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:21:59.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:21:59.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:21:59.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:21:59.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:21:59.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:21:59.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:59.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:21:59.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:21:59.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:59.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:21:59.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T20:21:59.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-recovery-scrub 2026-02-22T20:21:59.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:21:59.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:59.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.31002 2026-02-22T20:21:59.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.31002 2026-02-22T20:21:59.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:21:59.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:21:59.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-22T20:21:59.270 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T20:21:59.270 DEBUG:teuthology.orchestra.run.trial150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T20:21:59.319 INFO:tasks.workunit:Running workunit scrub/osd-scrub-dump.sh... 2026-02-22T20:21:59.320 DEBUG:teuthology.orchestra.run.trial150:workunit test scrub/osd-scrub-dump.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5c3084f688b1f6eff223b8d259a8316f9844c8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh 2026-02-22T20:21:59.372 INFO:tasks.workunit.client.0.trial150.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T20:21:59.374 INFO:tasks.workunit.client.0.trial150.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T20:21:59.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-scrub-dump 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:30: run: echo 'This test is disabled' 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stdout:This test is disabled 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-dump.sh:31: run: return 0 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-scrub-dump 0 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-dump 2026-02-22T20:21:59.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-22T20:21:59.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-dump KILL 2026-02-22T20:21:59.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:21:59.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:21:59.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:21:59.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:21:59.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:21:59.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:21:59.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:21:59.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:21:59.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:21:59.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:21:59.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:21:59.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:21:59.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:59.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:21:59.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:21:59.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:59.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:21:59.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T20:21:59.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-dump 2026-02-22T20:21:59.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:21:59.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:59.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40049 2026-02-22T20:21:59.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40049 2026-02-22T20:21:59.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:21:59.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:21:59.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-22T20:21:59.387 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T20:21:59.388 DEBUG:teuthology.orchestra.run.trial150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T20:21:59.442 INFO:tasks.workunit:Running workunit scrub/osd-scrub-repair.sh... 2026-02-22T20:21:59.442 DEBUG:teuthology.orchestra.run.trial150:workunit test scrub/osd-scrub-repair.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5c3084f688b1f6eff223b8d259a8316f9844c8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh 2026-02-22T20:21:59.490 INFO:tasks.workunit.client.0.trial150.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh 2026-02-22T20:21:59.491 INFO:tasks.workunit.client.0.trial150.stderr:++ TIMEOUT=300 2026-02-22T20:21:59.491 INFO:tasks.workunit.client.0.trial150.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2026-02-22T20:21:59.491 INFO:tasks.workunit.client.0.trial150.stderr:++ MAX_TIMEOUT=15 2026-02-22T20:21:59.491 INFO:tasks.workunit.client.0.trial150.stderr:++ PG_NUM=4 2026-02-22T20:21:59.491 INFO:tasks.workunit.client.0.trial150.stderr:++ TMPDIR=/tmp 2026-02-22T20:21:59.491 INFO:tasks.workunit.client.0.trial150.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2026-02-22T20:21:59.491 INFO:tasks.workunit.client.0.trial150.stderr:++ TESTDIR=/home/ubuntu/cephtest 2026-02-22T20:21:59.491 INFO:tasks.workunit.client.0.trial150.stderr:+++ uname 2026-02-22T20:21:59.492 INFO:tasks.workunit.client.0.trial150.stderr:++ '[' Linux = FreeBSD ']' 2026-02-22T20:21:59.492 INFO:tasks.workunit.client.0.trial150.stderr:++ SED=sed 2026-02-22T20:21:59.492 INFO:tasks.workunit.client.0.trial150.stderr:++ AWK=awk 2026-02-22T20:21:59.492 INFO:tasks.workunit.client.0.trial150.stderr:+++ stty -a 2026-02-22T20:21:59.492 INFO:tasks.workunit.client.0.trial150.stderr:+++ head -1 2026-02-22T20:21:59.492 INFO:tasks.workunit.client.0.trial150.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2026-02-22T20:21:59.492 INFO:tasks.workunit.client.0.trial150.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T20:21:59.493 INFO:tasks.workunit.client.0.trial150.stderr:++ termwidth= 2026-02-22T20:21:59.493 INFO:tasks.workunit.client.0.trial150.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2026-02-22T20:21:59.493 INFO:tasks.workunit.client.0.trial150.stderr:++ DIFFCOLOPTS='-y ' 2026-02-22T20:21:59.493 INFO:tasks.workunit.client.0.trial150.stderr:++ KERNCORE=kernel.core_pattern 2026-02-22T20:21:59.493 INFO:tasks.workunit.client.0.trial150.stderr:++ EXTRA_OPTS= 2026-02-22T20:21:59.494 INFO:tasks.workunit.client.0.trial150.stderr:++ test '' = TESTS 2026-02-22T20:21:59.494 INFO:tasks.workunit.client.0.trial150.stderr:+ source /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh 2026-02-22T20:21:59.495 INFO:tasks.workunit.client.0.trial150.stderr:++ uname 2026-02-22T20:21:59.496 INFO:tasks.workunit.client.0.trial150.stderr:+ '[' Linux = FreeBSD ']' 2026-02-22T20:21:59.496 INFO:tasks.workunit.client.0.trial150.stderr:+ use_ec_overwrite=true 2026-02-22T20:21:59.496 INFO:tasks.workunit.client.0.trial150.stderr:+ getjson=no 2026-02-22T20:21:59.496 INFO:tasks.workunit.client.0.trial150.stderr:+ jqfilter='def walk(f): 2026-02-22T20:21:59.497 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:21:59.497 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:21:59.497 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:21:59.497 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:21:59.497 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:21:59.497 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:21:59.497 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:21:59.497 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:21:59.497 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:21:59.497 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:21:59.497 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:21:59.498 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:21:59.498 INFO:tasks.workunit.client.0.trial150.stderr:+ sortkeys='import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:21:59.503 INFO:tasks.workunit.client.0.trial150.stderr:+ main osd-scrub-repair 2026-02-22T20:21:59.503 INFO:tasks.workunit.client.0.trial150.stderr:+ local dir=td/osd-scrub-repair 2026-02-22T20:21:59.503 INFO:tasks.workunit.client.0.trial150.stderr:+ shift 2026-02-22T20:21:59.503 INFO:tasks.workunit.client.0.trial150.stderr:+ shopt -s -o xtrace 2026-02-22T20:21:59.503 INFO:tasks.workunit.client.0.trial150.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T20:21:59.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T20:21:59.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T20:21:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-22T20:21:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-22T20:21:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-22T20:21:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-22T20:21:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-22T20:21:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-22T20:21:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-scrub-repair 2026-02-22T20:21:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:52: run: local dir=td/osd-scrub-repair 2026-02-22T20:21:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:53: run: shift 2026-02-22T20:21:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: export CEPH_MON=127.0.0.1:7107 2026-02-22T20:21:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:55: run: CEPH_MON=127.0.0.1:7107 2026-02-22T20:21:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:56: run: export CEPH_ARGS 2026-02-22T20:21:59.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: uuidgen 2026-02-22T20:21:59.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:57: run: CEPH_ARGS+='--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none ' 2026-02-22T20:21:59.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:58: run: CEPH_ARGS+='--mon-host=127.0.0.1:7107 ' 2026-02-22T20:21:59.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:59: run: CEPH_ARGS+='--osd-skip-data-digest=false ' 2026-02-22T20:21:59.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:61: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-22T20:21:59.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: set 2026-02-22T20:21:59.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T20:21:59.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:62: run: local 'funcs=TEST_allow_oper_initiated_scrub_during_recovery 2026-02-22T20:21:59.509 INFO:tasks.workunit.client.0.trial150.stderr:TEST_allow_repair_during_recovery 2026-02-22T20:21:59.509 INFO:tasks.workunit.client.0.trial150.stderr:TEST_auto_repair_bluestore_basic 2026-02-22T20:21:59.509 INFO:tasks.workunit.client.0.trial150.stderr:TEST_auto_repair_bluestore_failed 2026-02-22T20:21:59.509 INFO:tasks.workunit.client.0.trial150.stderr:TEST_auto_repair_bluestore_failed_norecov 2026-02-22T20:21:59.509 INFO:tasks.workunit.client.0.trial150.stderr:TEST_auto_repair_bluestore_scrub 2026-02-22T20:21:59.509 INFO:tasks.workunit.client.0.trial150.stderr:TEST_auto_repair_bluestore_tag 2026-02-22T20:21:59.510 INFO:tasks.workunit.client.0.trial150.stderr:TEST_auto_repair_erasure_coded_appends 2026-02-22T20:21:59.510 INFO:tasks.workunit.client.0.trial150.stderr:TEST_auto_repair_erasure_coded_overwrites 2026-02-22T20:21:59.510 INFO:tasks.workunit.client.0.trial150.stderr:TEST_corrupt_and_repair_jerasure_appends 2026-02-22T20:21:59.510 INFO:tasks.workunit.client.0.trial150.stderr:TEST_corrupt_and_repair_jerasure_overwrites 2026-02-22T20:21:59.510 INFO:tasks.workunit.client.0.trial150.stderr:TEST_corrupt_and_repair_lrc_appends 2026-02-22T20:21:59.510 INFO:tasks.workunit.client.0.trial150.stderr:TEST_corrupt_and_repair_lrc_overwrites 2026-02-22T20:21:59.510 INFO:tasks.workunit.client.0.trial150.stderr:TEST_corrupt_and_repair_replicated 2026-02-22T20:21:59.510 INFO:tasks.workunit.client.0.trial150.stderr:TEST_corrupt_scrub_erasure_appends 2026-02-22T20:21:59.510 INFO:tasks.workunit.client.0.trial150.stderr:TEST_corrupt_scrub_erasure_overwrites 2026-02-22T20:21:59.510 INFO:tasks.workunit.client.0.trial150.stderr:TEST_corrupt_scrub_replicated 2026-02-22T20:21:59.510 INFO:tasks.workunit.client.0.trial150.stderr:TEST_corrupt_snapset_scrub_rep 2026-02-22T20:21:59.510 INFO:tasks.workunit.client.0.trial150.stderr:TEST_dual_store_replicated_cluster 2026-02-22T20:21:59.511 INFO:tasks.workunit.client.0.trial150.stderr:TEST_list_missing_erasure_coded_appends 2026-02-22T20:21:59.511 INFO:tasks.workunit.client.0.trial150.stderr:TEST_list_missing_erasure_coded_overwrites 2026-02-22T20:21:59.511 INFO:tasks.workunit.client.0.trial150.stderr:TEST_periodic_scrub_replicated 2026-02-22T20:21:59.511 INFO:tasks.workunit.client.0.trial150.stderr:TEST_repair_stats 2026-02-22T20:21:59.511 INFO:tasks.workunit.client.0.trial150.stderr:TEST_repair_stats_ec 2026-02-22T20:21:59.511 INFO:tasks.workunit.client.0.trial150.stderr:TEST_request_scrub_priority 2026-02-22T20:21:59.511 INFO:tasks.workunit.client.0.trial150.stderr:TEST_scrub_warning 2026-02-22T20:21:59.511 INFO:tasks.workunit.client.0.trial150.stderr:TEST_skip_non_repair_during_recovery 2026-02-22T20:21:59.511 INFO:tasks.workunit.client.0.trial150.stderr:TEST_unfound_erasure_coded_appends 2026-02-22T20:21:59.511 INFO:tasks.workunit.client.0.trial150.stderr:TEST_unfound_erasure_coded_overwrites' 2026-02-22T20:21:59.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:21:59.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:21:59.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:21:59.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:21:59.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:21:59.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:21:59.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:21:59.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:21:59.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:21:59.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:21:59.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:21:59.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:21:59.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:21:59.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:21:59.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:21:59.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:21:59.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:21:59.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:21:59.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:21:59.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:59.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:21:59.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:21:59.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:21:59.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:21:59.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:21:59.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:21:59.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:21:59.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:59.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:21:59.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:21:59.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:21:59.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:21:59.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:21:59.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:21:59.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:21:59.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:21:59.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:21:59.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:21:59.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-22T20:21:59.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-22T20:21:59.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_oper_initiated_scrub_during_recovery td/osd-scrub-repair 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:115: TEST_allow_oper_initiated_scrub_during_recovery: local dir=td/osd-scrub-repair 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:116: TEST_allow_oper_initiated_scrub_during_recovery: cluster_conf=(['osds_num']='2' ['pgs_in_pool']='4' ['pool_name']='nopool' ['pool_default_size']='2' ['extras']='--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true') 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:116: TEST_allow_oper_initiated_scrub_during_recovery: local -A cluster_conf 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:125: TEST_allow_oper_initiated_scrub_during_recovery: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=2 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=nopool 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=nopool 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=2 2026-02-22T20:21:59.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=2 2026-02-22T20:21:59.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local 'extra_pars=--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true' 2026-02-22T20:21:59.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-22T20:21:59.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-22T20:21:59.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-22T20:21:59.812 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 c6c430af-5262-447f-804b-445eb2c065cf 2026-02-22T20:21:59.914 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:21:59.944 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:59.937+0000 7effc7dc8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:59.945 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:59.937+0000 7effc7dc8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:59.946 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:59.937+0000 7effc7dc8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:21:59.946 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:59.941+0000 7effc7dc8a40 -1 bdev(0x5628ea0e0c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:21:59.946 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:21:59.941+0000 7effc7dc8a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:22:00.683 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:22:00.881 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:22:00.913 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:00.905+0000 7f5942d73a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:00.915 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:00.909+0000 7f5942d73a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:00.916 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:00.909+0000 7f5942d73a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:01.036 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:22:01.297 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:01.289+0000 7f5942d73a40 -1 Falling back to public interface 2026-02-22T20:22:01.893 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:01.885+0000 7f5942d73a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:22:02.197 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:22:03.360 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:22:03.522 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3545857555,v1:127.0.0.1:6803/3545857555] [v2:127.0.0.1:6804/3545857555,v1:127.0.0.1:6805/3545857555] exists,up c6c430af-5262-447f-804b-445eb2c065cf 2026-02-22T20:22:03.526 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 9c034042-61d8-4689-bcf2-260164530aaa 2026-02-22T20:22:03.698 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:22:03.729 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:03.721+0000 7f44b4312a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:03.730 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:03.721+0000 7f44b4312a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:03.731 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:03.725+0000 7f44b4312a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:03.732 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:03.725+0000 7f44b4312a40 -1 bdev(0x562d95064c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:22:03.732 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:03.725+0000 7f44b4312a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:22:04.407 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:22:04.608 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:22:04.640 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:04.633+0000 7f0841c36a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:04.641 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:04.633+0000 7f0841c36a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:04.642 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:04.633+0000 7f0841c36a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:04.769 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:22:05.040 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:05.033+0000 7f0841c36a40 -1 Falling back to public interface 2026-02-22T20:22:05.585 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:05.577+0000 7f0841c36a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:22:05.927 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:22:07.084 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:22:07.236 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4156205037,v1:127.0.0.1:6811/4156205037] [v2:127.0.0.1:6812/4156205037,v1:127.0.0.1:6813/4156205037] exists,up 9c034042-61d8-4689-bcf2-260164530aaa 2026-02-22T20:22:08.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:126: TEST_allow_oper_initiated_scrub_during_recovery: local poolname=rbd 2026-02-22T20:22:08.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:127: TEST_allow_oper_initiated_scrub_during_recovery: create_rbd_pool 2026-02-22T20:22:08.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:22:08.386 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:22:08.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:22:08.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:22:08.596 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:22:08.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:22:09.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:22:09.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:128: TEST_allow_oper_initiated_scrub_during_recovery: wait_for_clean 2026-02-22T20:22:09.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:22:09.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:22:09.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:22:09.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:22:09.941 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:22:09.941 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:22:09.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:22:09.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:22:09.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:22:10.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:22:10.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:22:10.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:22:10.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:22:10.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:22:10.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:22:10.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:22:10.204 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:22:10.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:22:10.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:22:10.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:22:10.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-22T20:22:10.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-22T20:22:10.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-22T20:22:10.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:22:10.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:22:10.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-22T20:22:10.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-22T20:22:10.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672965' 2026-02-22T20:22:10.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:22:10.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-22T20:22:10.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:22:10.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:22:10.352 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-22T20:22:10.353 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:22:10.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-22T20:22:10.354 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836489 2026-02-22T20:22:10.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-22T20:22:10.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:22:10.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-22T20:22:10.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:22:10.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-22T20:22:10.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:22:10.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:22:10.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-22T20:22:10.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:22:10.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-22T20:22:10.521 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672965 2026-02-22T20:22:10.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-22T20:22:10.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:22:10.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672965 2026-02-22T20:22:10.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:22:11.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:22:11.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:22:11.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672965 2026-02-22T20:22:11.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:22:12.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:22:12.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:22:13.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672967 -lt 42949672965 2026-02-22T20:22:13.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:22:13.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:22:13.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:22:13.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:22:13.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:22:13.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:22:13.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:22:13.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:22:13.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:22:13.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:22:13.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:22:13.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:22:13.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:22:13.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:22:13.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:22:13.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:22:13.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:22:13.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:22:13.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:130: TEST_allow_oper_initiated_scrub_during_recovery: add_something td/osd-scrub-repair rbd 2026-02-22T20:22:13.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:22:13.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2026-02-22T20:22:13.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-22T20:22:13.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:22:13.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:22:13.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:22:13.755 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:22:13.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:22:13.960 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:22:13.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:22:13.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:22:13.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:22:13.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:131: TEST_allow_oper_initiated_scrub_during_recovery: get_not_primary rbd SOMETHING 2026-02-22T20:22:13.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-22T20:22:13.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:22:13.992 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2026-02-22T20:22:13.992 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-22T20:22:13.992 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:22:13.993 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-22T20:22:13.993 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:22:14.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:22:14.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2026-02-22T20:22:14.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:22:14.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:131: TEST_allow_oper_initiated_scrub_during_recovery: oper_scrub_and_schedule td/osd-scrub-repair rbd 0 2026-02-22T20:22:14.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:180: oper_scrub_and_schedule: local dir=td/osd-scrub-repair 2026-02-22T20:22:14.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:181: oper_scrub_and_schedule: local poolname=rbd 2026-02-22T20:22:14.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:182: oper_scrub_and_schedule: local osd=0 2026-02-22T20:22:14.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: oper_scrub_and_schedule: get_pg rbd SOMETHING 2026-02-22T20:22:14.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-22T20:22:14.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:22:14.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-22T20:22:14.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:22:14.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:187: oper_scrub_and_schedule: local pg=1.3 2026-02-22T20:22:14.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: oper_scrub_and_schedule: get_last_scrub_stamp 1.3 2026-02-22T20:22:14.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:22:14.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:22:14.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:22:14.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:22:14.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:188: oper_scrub_and_schedule: local last_scrub=2026-02-22T20:22:08.592416+0000 2026-02-22T20:22:14.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:189: oper_scrub_and_schedule: ceph tell 1.3 scrub 2026-02-22T20:22:14.707 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:22:14.707 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:22:14.707 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:22:14.707 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:22:14.707 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:22:14.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:194: oper_scrub_and_schedule: sleep 3 2026-02-22T20:22:17.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: oper_scrub_and_schedule: (( i=0 )) 2026-02-22T20:22:17.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:195: oper_scrub_and_schedule: (( i < 3 )) 2026-02-22T20:22:17.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: oper_scrub_and_schedule: get_last_scrub_stamp 1.3 2026-02-22T20:22:17.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:22:17.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:22:17.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:22:17.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:22:17.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:196: oper_scrub_and_schedule: test 2026-02-22T20:22:14.712430+0000 '>' 2026-02-22T20:22:08.592416+0000 2026-02-22T20:22:17.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:197: oper_scrub_and_schedule: break 2026-02-22T20:22:17.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:208: oper_scrub_and_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:22:17.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:22:17.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:22:17.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:22:17.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:22:17.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:22:17.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:22:17.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:22:17.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:22:17.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:22:17.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:22:17.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:22:17.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:22:17.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:22:17.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:22:17.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:22:18.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:22:18.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:22:18.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:22:18.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:22:18.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:22:18.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:22:18.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:22:18.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-22T20:22:18.266 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:22:18.266 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:22:18.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:22:18.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:22:18.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:22:18.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:22:18.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:22:18.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:22:18.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:22:18.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:22:18.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:22:18.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:22:18.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:22:18.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:22:18.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:22:18.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:22:18.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:22:18.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:22:18.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:22:18.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:22:18.376 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:22:18.376 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:18.376 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:18.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:22:18.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:22:18.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:22:18.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:22:18.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:22:18.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:22:18.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:22:18.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:22:18.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:22:18.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:22:18.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:22:18.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:22:18.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:22:18.378 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:22:18.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:22:18.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:22:18.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:22:18.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:22:18.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:22:18.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:22:18.394 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:18.385+0000 7fdb07b2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:18.396 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:18.389+0000 7fdb07b2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:18.396 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:18.389+0000 7fdb07b2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:18.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:22:18.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:22:18.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:22:18.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:22:18.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:22:18.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:22:18.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:22:18.540 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:22:18.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:22:18.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:22:18.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:22:18.845 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:18.837+0000 7fdb07b2aa40 -1 Falling back to public interface 2026-02-22T20:22:19.512 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:19.505+0000 7fdb07b2aa40 -1 osd.0 20 log_to_monitors true 2026-02-22T20:22:19.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:22:19.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:22:19.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:22:19.701 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:22:19.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:22:19.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:22:19.857 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [5,20) [v2:127.0.0.1:6802/2755845428,v1:127.0.0.1:6803/2755845428] [v2:127.0.0.1:6804/2755845428,v1:127.0.0.1:6805/2755845428] exists,up c6c430af-5262-447f-804b-445eb2c065cf 2026-02-22T20:22:19.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:22:19.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:22:19.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:22:19.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:22:19.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:22:19.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:22:19.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:22:19.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:22:19.858 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:22:19.858 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:22:19.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:22:19.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:22:19.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:22:19.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:22:19.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:22:19.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:22:19.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:22:19.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:22:19.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:22:20.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:22:20.128 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:22:20.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:22:20.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:22:20.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:22:20.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-22T20:22:20.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-22T20:22:20.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-22T20:22:20.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:22:20.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:22:20.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672976 2026-02-22T20:22:20.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672976 2026-02-22T20:22:20.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-42949672976' 2026-02-22T20:22:20.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:22:20.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-22T20:22:20.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:22:20.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:22:20.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-22T20:22:20.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:22:20.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-22T20:22:20.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-22T20:22:20.280 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 98784247810 2026-02-22T20:22:20.280 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:22:20.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247810 2026-02-22T20:22:20.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:22:20.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672976 2026-02-22T20:22:20.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:22:20.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:22:20.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672976 2026-02-22T20:22:20.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:22:20.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672976 2026-02-22T20:22:20.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2026-02-22T20:22:20.442 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672976 2026-02-22T20:22:20.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:22:20.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672976 -lt 42949672976 2026-02-22T20:22:20.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:22:20.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:22:20.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:22:20.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:22:20.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:22:20.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:22:20.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:22:20.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:22:20.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:22:20.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:22:20.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:22:20.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:22:20.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:22:20.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:22:20.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:22:21.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:22:21.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:22:21.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:22:21.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:209: oper_scrub_and_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:22:21.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:210: oper_scrub_and_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:22:21.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:22:21.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:22:21.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:22:21.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:22:21.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:22:21.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:22:21.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:22:21.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:22:21.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:22:21.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:22:21.280 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:22:21.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:22:21.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:22:21.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:22:21.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:22:21.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:22:21.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:22:21.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:22:21.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:22:21.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:22:21.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:22:21.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:22:21.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:22:21.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:22:21.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:21.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:21.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:22:21.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:22:21.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:22:21.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:22:21.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:22:21.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:22:21.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:22:21.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:22:21.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:22:21.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:22:21.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:22:21.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:22:21.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:22:21.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:22:21.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:22:21.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:22:21.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:22:21.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:22:21.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:22:21.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:22:21.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:22:21.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:22:21.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:22:21.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:22:21.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:22:21.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:22:21.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:22:21.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:22:21.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:22:21.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:22:21.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:21.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:21.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:22:21.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:22:21.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:22:21.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:22:21.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:22:21.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:21.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:21.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:22:21.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_allow_repair_during_recovery td/osd-scrub-repair 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:138: TEST_allow_repair_during_recovery: local dir=td/osd-scrub-repair 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:139: TEST_allow_repair_during_recovery: local poolname=rbd 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:141: TEST_allow_repair_during_recovery: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:22:21.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-22T20:22:21.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:22:21.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:22:21.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:22:21.337 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:22:21.337 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:21.337 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:21.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:22:21.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-22T20:22:21.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:22:21.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:22:21.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:22:21.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:22:21.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:22:21.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:22:21.379 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:22:21.379 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:22:21.379 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:22:21.379 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:22:21.379 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:21.379 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:21.380 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:22:21.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:22:21.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:22:21.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:22:21.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:22:21.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:22:21.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:22:21.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:22:21.436 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:22:21.437 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:22:21.437 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:22:21.437 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:22:21.437 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:21.437 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:21.437 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:22:21.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:22:21.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:22:21.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:142: TEST_allow_repair_during_recovery: run_mgr td/osd-scrub-repair x 2026-02-22T20:22:21.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:22:21.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:22:21.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:22:21.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:22:21.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:22:21.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:22:21.590 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:22:21.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:22:21.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:22:21.591 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:22:21.591 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:21.591 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:21.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:22:21.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:22:21.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:22:21.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:143: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 0 --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true 2026-02-22T20:22:21.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:22:21.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:22:21.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:22:21.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:22:21.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:22:21.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:22:21.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:22:21.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:22:21.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:22:21.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:22:21.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:22:21.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:22:21.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:22:21.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:22:21.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:22:21.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:22:21.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:22:21.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:22:21.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true' 2026-02-22T20:22:21.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:22:21.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:22:21.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2e5c4838-2e64-4502-a3d2-0e0f05359284 2026-02-22T20:22:21.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2e5c4838-2e64-4502-a3d2-0e0f05359284' 2026-02-22T20:22:21.616 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 2e5c4838-2e64-4502-a3d2-0e0f05359284 2026-02-22T20:22:21.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:22:21.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB9ZZtpK1wLJRAAroKsKE7o0QKWOPqAa1qn8Q== 2026-02-22T20:22:21.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB9ZZtpK1wLJRAAroKsKE7o0QKWOPqAa1qn8Q=="}' 2026-02-22T20:22:21.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2e5c4838-2e64-4502-a3d2-0e0f05359284 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:22:21.712 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:22:21.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:22:21.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true --mkfs --key AQB9ZZtpK1wLJRAAroKsKE7o0QKWOPqAa1qn8Q== --osd-uuid 2e5c4838-2e64-4502-a3d2-0e0f05359284 2026-02-22T20:22:21.737 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:21.729+0000 7fc3ac274a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:21.738 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:21.729+0000 7fc3ac274a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:21.738 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:21.729+0000 7fc3ac274a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:21.739 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:21.729+0000 7fc3ac274a40 -1 bdev(0x556f93b6ec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:22:21.739 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:21.729+0000 7fc3ac274a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:22:22.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:22:22.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:22:22.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:22:22.507 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:22:22.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:22:22.703 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:22:22.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:22:22.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true 2026-02-22T20:22:22.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:22:22.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:22:22.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:22:22.719 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:22.709+0000 7f6957732a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:22.720 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:22.713+0000 7f6957732a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:22.721 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:22.713+0000 7f6957732a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:22.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:22:22.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:22:22.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:22:22.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:22:22.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:22:22.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:22:22.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:22:22.860 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:22:22.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:22:22.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:22:23.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:22:23.153 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:23.145+0000 7f6957732a40 -1 Falling back to public interface 2026-02-22T20:22:23.789 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:23.781+0000 7f6957732a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:22:24.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:22:24.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:22:24.022 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:22:24.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:22:24.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:22:24.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:22:24.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:22:25.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:22:25.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:22:25.183 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:22:25.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:22:25.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:22:25.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:22:25.341 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/885638666,v1:127.0.0.1:6803/885638666] [v2:127.0.0.1:6804/885638666,v1:127.0.0.1:6805/885638666] exists,up 2e5c4838-2e64-4502-a3d2-0e0f05359284 2026-02-22T20:22:25.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:22:25.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:22:25.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:22:25.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:145: TEST_allow_repair_during_recovery: run_osd td/osd-scrub-repair 1 --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true 2026-02-22T20:22:25.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:22:25.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:22:25.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:22:25.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:22:25.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:22:25.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:22:25.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:22:25.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:22:25.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:22:25.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:22:25.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:22:25.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:22:25.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:22:25.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:22:25.343 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:22:25.343 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true' 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:22:25.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:22:25.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=db8fc798-72b3-4198-afec-4d1fdeec4442 2026-02-22T20:22:25.346 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 db8fc798-72b3-4198-afec-4d1fdeec4442 2026-02-22T20:22:25.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 db8fc798-72b3-4198-afec-4d1fdeec4442' 2026-02-22T20:22:25.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:22:25.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCBZZtp/ZgbFRAAnDjeC08gcIxkUWuEZaNvgA== 2026-02-22T20:22:25.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCBZZtp/ZgbFRAAnDjeC08gcIxkUWuEZaNvgA=="}' 2026-02-22T20:22:25.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new db8fc798-72b3-4198-afec-4d1fdeec4442 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:22:25.523 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:22:25.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:22:25.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true --mkfs --key AQCBZZtp/ZgbFRAAnDjeC08gcIxkUWuEZaNvgA== --osd-uuid db8fc798-72b3-4198-afec-4d1fdeec4442 2026-02-22T20:22:25.550 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:25.541+0000 7fbec9e98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:25.551 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:25.545+0000 7fbec9e98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:25.552 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:25.545+0000 7fbec9e98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:25.553 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:25.545+0000 7fbec9e98a40 -1 bdev(0x55999a438c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:22:25.553 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:25.545+0000 7fbec9e98a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:22:26.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:22:26.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:22:26.224 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:22:26.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:22:26.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:22:26.428 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:22:26.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:22:26.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true 2026-02-22T20:22:26.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:22:26.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:22:26.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:22:26.453 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:26.445+0000 7faa49bdda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:26.455 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:26.445+0000 7faa49bdda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:26.456 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:26.449+0000 7faa49bdda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:26.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:22:26.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:22:26.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:22:26.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:22:26.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:22:26.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:22:26.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:22:26.587 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:22:26.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:22:26.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:22:26.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:22:26.781 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:26.773+0000 7faa49bdda40 -1 Falling back to public interface 2026-02-22T20:22:27.338 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:27.329+0000 7faa49bdda40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:22:27.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:22:27.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:22:27.747 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:22:27.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:22:27.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:22:27.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:22:27.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:22:28.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:22:28.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:22:28.906 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:22:28.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:22:28.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:22:28.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:22:29.064 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/870932415,v1:127.0.0.1:6811/870932415] [v2:127.0.0.1:6812/870932415,v1:127.0.0.1:6813/870932415] exists,up db8fc798-72b3-4198-afec-4d1fdeec4442 2026-02-22T20:22:29.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:22:29.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:22:29.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:22:29.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:147: TEST_allow_repair_during_recovery: create_rbd_pool 2026-02-22T20:22:29.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:22:29.217 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:22:29.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:22:29.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:22:29.448 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:22:29.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:22:30.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:22:30.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:148: TEST_allow_repair_during_recovery: wait_for_clean 2026-02-22T20:22:30.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:22:30.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:22:30.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:22:30.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:22:30.702 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:22:30.702 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:22:30.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:22:30.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:22:30.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:22:30.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:22:30.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:22:30.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:22:30.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:22:30.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:22:30.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:22:30.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:22:30.965 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:22:30.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:22:30.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:22:30.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:22:31.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:22:31.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:22:31.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:22:31.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:22:31.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:22:31.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-22T20:22:31.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-22T20:22:31.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-22T20:22:31.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:22:31.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:22:31.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:22:31.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:22:31.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:22:31.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:22:31.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:22:31.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:22:31.121 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:22:31.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:22:31.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:22:31.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:22:32.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:22:32.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:22:32.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T20:22:32.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:22:32.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-22T20:22:32.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:22:32.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:22:32.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-22T20:22:32.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:22:32.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-22T20:22:32.454 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705667 2026-02-22T20:22:32.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-22T20:22:32.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:22:32.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-22T20:22:32.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:22:32.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:22:32.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:22:32.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:22:32.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:22:32.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:22:32.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:22:32.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:22:32.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:22:32.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:22:32.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:22:32.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:22:32.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:22:32.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:22:32.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:22:33.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:22:33.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:22:33.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:22:33.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:150: TEST_allow_repair_during_recovery: add_something td/osd-scrub-repair rbd 2026-02-22T20:22:33.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:22:33.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2026-02-22T20:22:33.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-22T20:22:33.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:22:33.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:22:33.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:22:33.363 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:22:33.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:22:33.566 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:22:33.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:22:33.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:22:33.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:22:33.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_allow_repair_during_recovery: get_not_primary rbd SOMETHING 2026-02-22T20:22:33.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-22T20:22:33.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:22:33.602 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2026-02-22T20:22:33.602 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-22T20:22:33.602 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:22:33.602 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-22T20:22:33.603 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:22:33.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:22:33.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2026-02-22T20:22:33.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:22:33.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:151: TEST_allow_repair_during_recovery: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2026-02-22T20:22:33.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-22T20:22:33.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=rbd 2026-02-22T20:22:33.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=0 2026-02-22T20:22:33.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:22:33.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:22:33.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:22:33.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:22:33.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:22:33.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:22:33.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:22:33.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:22:33.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:22:33.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:22:33.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:22:33.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:22:33.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:22:33.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:22:33.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:22:33.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:22:34.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:22:34.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:22:34.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:22:34.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:22:34.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:22:34.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:22:34.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:22:34.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-22T20:22:34.442 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-22T20:22:34.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:22:34.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:22:34.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:22:34.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:22:34.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:22:34.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:34.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:22:34.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:22:34.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:22:34.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:22:34.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:22:34.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:22:34.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:22:34.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:22:34.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:22:34.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:22:34.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:22:34.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:22:34.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:22:34.535 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:22:34.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:22:34.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:22:34.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:22:34.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:22:34.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:22:34.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:22:34.554 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:34.544+0000 7fe124177a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:34.556 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:34.548+0000 7fe124177a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:34.557 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:34.548+0000 7fe124177a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:34.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:22:34.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:22:34.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:22:34.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:22:34.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:22:34.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:22:34.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:22:34.694 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:22:34.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:22:34.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:22:34.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:22:34.929 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:34.920+0000 7fe124177a40 -1 Falling back to public interface 2026-02-22T20:22:35.515 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:35.504+0000 7fe124177a40 -1 osd.0 18 log_to_monitors true 2026-02-22T20:22:35.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:22:35.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:22:35.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:22:35.855 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:22:35.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:22:35.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:22:36.008 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 22 up_thru 22 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6802/3667921364,v1:127.0.0.1:6803/3667921364] [v2:127.0.0.1:6804/3667921364,v1:127.0.0.1:6805/3667921364] exists,up 2e5c4838-2e64-4502-a3d2-0e0f05359284 2026-02-22T20:22:36.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:22:36.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:22:36.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:22:36.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:22:36.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:22:36.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:22:36.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:22:36.009 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:22:36.010 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:22:36.010 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:22:36.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:22:36.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:22:36.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:22:36.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:22:36.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:22:36.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:22:36.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:22:36.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:22:36.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:22:36.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:22:36.275 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:22:36.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:22:36.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:22:36.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:22:36.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280514 2026-02-22T20:22:36.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280514 2026-02-22T20:22:36.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514' 2026-02-22T20:22:36.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:22:36.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:22:36.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-22T20:22:36.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-22T20:22:36.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514 1-38654705669' 2026-02-22T20:22:36.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:22:36.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280514 2026-02-22T20:22:36.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:22:36.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:22:36.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280514 2026-02-22T20:22:36.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:22:36.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280514 2026-02-22T20:22:36.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280514' 2026-02-22T20:22:36.423 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 94489280514 2026-02-22T20:22:36.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:22:36.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 94489280514 2026-02-22T20:22:36.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:22:37.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:22:37.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:22:37.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 94489280514 2026-02-22T20:22:37.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:22:38.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:22:38.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:22:38.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280514 -lt 94489280514 2026-02-22T20:22:38.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:22:38.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-22T20:22:38.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:22:38.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:22:38.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-22T20:22:38.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:22:38.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-22T20:22:38.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-22T20:22:38.910 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 38654705669 2026-02-22T20:22:38.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:22:39.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-22T20:22:39.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:22:39.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:22:39.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:22:39.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:22:39.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:22:39.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:22:39.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:22:39.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:22:39.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:22:39.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:22:39.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:22:39.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:22:39.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:22:39.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:22:39.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:22:39.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:22:39.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:22:39.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:22:39.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg rbd SOMETHING 2026-02-22T20:22:39.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-22T20:22:39.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:22:39.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-22T20:22:39.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:22:39.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=1.3 2026-02-22T20:22:39.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 1.3 2026-02-22T20:22:39.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.3 2026-02-22T20:22:39.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.3 2026-02-22T20:22:39.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:22:39.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:22:39.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:22:39.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:22:39.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:22:29.443604+0000 2026-02-22T20:22:39.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.3 2026-02-22T20:22:40.086 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.3 on osd.1 to repair 2026-02-22T20:22:40.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.3 2026-02-22T20:22:29.443604+0000 2026-02-22T20:22:40.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.3 2026-02-22T20:22:40.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:22:29.443604+0000 2026-02-22T20:22:40.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:22:40.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:22:40.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:22:40.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:22:40.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:22:40.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:22:40.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:22:40.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:22:40.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:22:29.443604+0000 '>' 2026-02-22T20:22:29.443604+0000 2026-02-22T20:22:40.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:22:41.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:22:41.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:22:41.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:22:41.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:22:41.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:22:41.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:22:41.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:22:41.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:22:29.443604+0000 '>' 2026-02-22T20:22:29.443604+0000 2026-02-22T20:22:41.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:22:42.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:22:42.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:22:42.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:22:42.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:22:42.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:22:42.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:22:42.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:22:42.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:22:29.443604+0000 '>' 2026-02-22T20:22:29.443604+0000 2026-02-22T20:22:42.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:22:43.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:22:43.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:22:43.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:22:43.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:22:43.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:22:43.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:22:43.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:22:43.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:22:29.443604+0000 '>' 2026-02-22T20:22:29.443604+0000 2026-02-22T20:22:43.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:22:44.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:22:44.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:22:44.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:22:44.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:22:44.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:22:44.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:22:44.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:22:44.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:22:40.427598+0000 '>' 2026-02-22T20:22:29.443604+0000 2026-02-22T20:22:44.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:22:44.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:22:44.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:22:44.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:22:44.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:22:44.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:22:44.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:22:44.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:22:44.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:22:44.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:22:44.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:22:44.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:22:44.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:22:44.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:22:44.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:22:44.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:22:44.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:22:45.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:22:45.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:22:45.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:22:45.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:22:45.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:22:45.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:22:45.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:22:45.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-22T20:22:45.286 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:22:45.286 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:22:45.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:22:45.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:22:45.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:22:45.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:22:45.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:22:45.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:22:45.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:22:45.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:22:45.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:22:45.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:22:45.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:22:45.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:22:45.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:22:45.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:22:45.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:22:45.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:22:45.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:22:45.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:22:45.370 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:22:45.370 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:45.370 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:45.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:22:45.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:22:45.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:22:45.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:22:45.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:22:45.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:22:45.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:22:45.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:22:45.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:22:45.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:22:45.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:22:45.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:22:45.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:22:45.371 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:22:45.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:22:45.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:22:45.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:22:45.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:22:45.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:22:45.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:22:45.389 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:45.380+0000 7f604c398a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:45.390 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:45.380+0000 7f604c398a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:45.391 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:45.380+0000 7f604c398a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:45.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:22:45.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:22:45.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:22:45.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:22:45.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:22:45.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:22:45.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:22:45.531 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:22:45.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:22:45.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:22:45.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:22:45.765 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:45.756+0000 7f604c398a40 -1 Falling back to public interface 2026-02-22T20:22:46.371 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:46.364+0000 7f604c398a40 -1 osd.1 27 log_to_monitors true 2026-02-22T20:22:46.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:22:46.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:22:46.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:22:46.688 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:22:46.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:22:46.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:22:46.850 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [9,27) [v2:127.0.0.1:6810/4143288718,v1:127.0.0.1:6811/4143288718] [v2:127.0.0.1:6812/4143288718,v1:127.0.0.1:6813/4143288718] exists,up db8fc798-72b3-4198-afec-4d1fdeec4442 2026-02-22T20:22:46.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:22:46.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:22:46.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:22:46.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:22:46.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:22:46.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:22:46.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:22:46.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:22:46.852 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:22:46.852 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:22:46.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:22:46.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:22:46.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:22:46.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:22:46.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:22:46.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:22:46.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:22:46.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:22:46.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:22:47.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:22:47.114 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:22:47.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:22:47.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:22:47.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:22:47.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280517 2026-02-22T20:22:47.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280517 2026-02-22T20:22:47.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280517' 2026-02-22T20:22:47.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:22:47.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:22:47.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-22T20:22:47.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-22T20:22:47.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280517 1-128849018882' 2026-02-22T20:22:47.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:22:47.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280517 2026-02-22T20:22:47.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:22:47.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:22:47.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280517 2026-02-22T20:22:47.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:22:47.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280517 2026-02-22T20:22:47.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280517' 2026-02-22T20:22:47.262 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 94489280517 2026-02-22T20:22:47.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:22:47.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280516 -lt 94489280517 2026-02-22T20:22:47.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:22:48.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:22:48.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:22:48.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280517 -lt 94489280517 2026-02-22T20:22:48.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:22:48.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-128849018882 2026-02-22T20:22:48.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:22:48.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:22:48.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-128849018882 2026-02-22T20:22:48.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:22:48.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-22T20:22:48.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2026-02-22T20:22:48.583 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 128849018882 2026-02-22T20:22:48.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:22:48.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018882 -lt 128849018882 2026-02-22T20:22:48.749 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:22:48.749 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:22:48.749 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:22:48.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:22:48.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:22:48.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:22:48.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:22:48.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:22:48.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:22:48.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:22:48.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:22:49.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:22:49.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:22:49.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:22:49.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:22:49.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:22:49.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:22:49.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:22:49.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:22:49.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:22:49.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:22:49.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:22:49.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:22:49.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:22:49.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:22:49.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:22:49.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:22:49.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:22:49.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:22:49.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:22:49.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:22:49.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:22:49.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:22:49.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:22:49.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:22:49.428 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:22:49.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:22:49.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:22:49.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:22:49.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:22:49.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:22:49.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:22:49.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:22:49.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:22:49.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:49.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:49.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:22:49.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:22:49.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:22:49.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:22:49.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:22:49.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:22:49.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:22:49.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:22:49.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:22:49.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:22:49.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:22:49.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:22:49.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:22:49.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:22:49.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:22:49.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:22:49.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:22:49.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:22:49.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:22:49.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:22:49.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:22:49.449 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:22:49.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:22:49.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:22:49.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:22:49.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:22:49.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:22:49.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:22:49.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:22:49.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:22:49.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:49.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:49.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:22:49.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:22:49.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:22:49.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:22:49.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:22:49.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:22:49.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:22:49.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:22:49.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_basic td/osd-scrub-repair 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:498: TEST_auto_repair_bluestore_basic: local dir=td/osd-scrub-repair 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_basic: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='1' ['pool_name']='testpool' ['extras']=' --osd_scrub_auto_repair=true') 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:499: TEST_auto_repair_bluestore_basic: local -A cluster_conf 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:505: TEST_auto_repair_bluestore_basic: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=1 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=testpool 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=testpool 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=3 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=3 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local 'extra_pars= --osd_scrub_auto_repair=true' 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-22T20:22:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-22T20:22:49.749 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 fd4cbde9-c19a-402a-b381-fb310894d983 2026-02-22T20:22:49.851 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:22:49.881 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:49.872+0000 7f0f7df63a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:49.883 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:49.872+0000 7f0f7df63a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:49.884 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:49.876+0000 7f0f7df63a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:49.884 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:49.876+0000 7f0f7df63a40 -1 bdev(0x5584e35bec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:22:49.884 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:49.876+0000 7f0f7df63a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:22:50.747 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:22:50.938 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:22:50.971 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:50.960+0000 7f5955bb8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:50.972 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:50.964+0000 7f5955bb8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:50.973 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:50.964+0000 7f5955bb8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:51.100 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:22:51.329 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:51.320+0000 7f5955bb8a40 -1 Falling back to public interface 2026-02-22T20:22:51.926 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:51.916+0000 7f5955bb8a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:22:52.261 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:22:53.427 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:22:53.583 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3001442294,v1:127.0.0.1:6803/3001442294] [v2:127.0.0.1:6804/3001442294,v1:127.0.0.1:6805/3001442294] exists,up fd4cbde9-c19a-402a-b381-fb310894d983 2026-02-22T20:22:53.589 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 f8b1aaa0-fc0f-4cf3-83bd-8c462343bdc7 2026-02-22T20:22:53.760 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:22:53.793 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:53.784+0000 7f8b6464fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:53.794 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:53.784+0000 7f8b6464fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:53.795 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:53.784+0000 7f8b6464fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:53.796 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:53.784+0000 7f8b6464fa40 -1 bdev(0x55ee52742c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:22:53.796 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:53.788+0000 7f8b6464fa40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:22:54.392 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:22:54.593 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:22:54.624 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:54.616+0000 7fa29611ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:54.626 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:54.616+0000 7fa29611ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:54.627 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:54.616+0000 7fa29611ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:54.751 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:22:55.034 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:55.024+0000 7fa29611ba40 -1 Falling back to public interface 2026-02-22T20:22:55.678 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:55.668+0000 7fa29611ba40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:22:55.913 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:22:57.071 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:22:57.229 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/724996244,v1:127.0.0.1:6811/724996244] [v2:127.0.0.1:6812/724996244,v1:127.0.0.1:6813/724996244] exists,up f8b1aaa0-fc0f-4cf3-83bd-8c462343bdc7 2026-02-22T20:22:57.235 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 ae466ee6-e1c4-47fc-9ab2-a00cdc75febe 2026-02-22T20:22:57.406 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:22:57.437 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:57.428+0000 7f99b36d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:57.438 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:57.428+0000 7f99b36d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:57.439 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:57.428+0000 7f99b36d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:57.440 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:57.432+0000 7f99b36d0a40 -1 bdev(0x5611960cec00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:22:57.440 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:57.432+0000 7f99b36d0a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:22:58.070 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:22:58.273 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:22:58.305 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:58.296+0000 7f12ef930a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:58.306 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:58.296+0000 7f12ef930a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:58.307 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:58.296+0000 7f12ef930a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:22:58.434 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:22:58.729 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:58.720+0000 7f12ef930a40 -1 Falling back to public interface 2026-02-22T20:22:59.294 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:22:59.284+0000 7f12ef930a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:22:59.592 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:23:00.748 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:23:00.903 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/450483087,v1:127.0.0.1:6819/450483087] [v2:127.0.0.1:6820/450483087,v1:127.0.0.1:6821/450483087] exists,up ae466ee6-e1c4-47fc-9ab2-a00cdc75febe 2026-02-22T20:23:01.105 INFO:tasks.workunit.client.0.trial150.stderr:pool 'testpool' created 2026-02-22T20:23:02.599 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836491 2026-02-22T20:23:05.090 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672967 2026-02-22T20:23:05.261 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509444 2026-02-22T20:23:07.107 INFO:tasks.workunit.client.0.trial150.stdout:standard_scrub_cluster: dbg: test pool is testpool 1 2026-02-22T20:23:07.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:506: TEST_auto_repair_bluestore_basic: local poolid=1 2026-02-22T20:23:07.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:507: TEST_auto_repair_bluestore_basic: local poolname=testpool 2026-02-22T20:23:07.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:509: TEST_auto_repair_bluestore_basic: ceph osd pool set testpool size 2 2026-02-22T20:23:07.309 INFO:tasks.workunit.client.0.trial150.stderr:set pool 1 size to 2 2026-02-22T20:23:07.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:510: TEST_auto_repair_bluestore_basic: wait_for_clean 2026-02-22T20:23:07.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:23:07.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:23:07.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:23:07.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:23:07.324 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:23:07.324 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:23:07.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:23:07.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:23:07.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:23:07.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:23:07.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:23:07.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:23:07.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:23:07.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:23:07.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:23:07.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:23:07.595 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:23:07.595 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:23:07.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:23:07.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:07.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:23:07.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-22T20:23:07.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-22T20:23:07.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-22T20:23:07.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:07.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:23:07.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672973 2026-02-22T20:23:07.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672973 2026-02-22T20:23:07.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672973' 2026-02-22T20:23:07.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:07.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:23:07.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509450 2026-02-22T20:23:07.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509450 2026-02-22T20:23:07.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672973 2-64424509450' 2026-02-22T20:23:07.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:07.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-22T20:23:07.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:07.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:23:07.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-22T20:23:07.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:07.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-22T20:23:07.817 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836497 2026-02-22T20:23:07.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-22T20:23:07.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:07.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836497 2026-02-22T20:23:07.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:23:08.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:23:08.976 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:09.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836497 2026-02-22T20:23:09.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:09.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672973 2026-02-22T20:23:09.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:09.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:23:09.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672973 2026-02-22T20:23:09.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:09.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672973 2026-02-22T20:23:09.137 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672973 2026-02-22T20:23:09.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2026-02-22T20:23:09.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:23:09.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672974 -lt 42949672973 2026-02-22T20:23:09.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:09.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509450 2026-02-22T20:23:09.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:09.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:23:09.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509450 2026-02-22T20:23:09.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:09.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509450 2026-02-22T20:23:09.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2026-02-22T20:23:09.302 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509450 2026-02-22T20:23:09.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:23:09.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509450 -lt 64424509450 2026-02-22T20:23:09.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:23:09.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:09.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:09.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:23:09.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:23:09.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:23:09.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:23:09.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:23:09.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:23:09.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:23:09.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:23:09.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:23:09.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:23:09.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:09.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:09.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:23:09.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:23:09.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:23:09.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:513: TEST_auto_repair_bluestore_basic: local payload=ABCDEF 2026-02-22T20:23:09.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:514: TEST_auto_repair_bluestore_basic: echo ABCDEF 2026-02-22T20:23:09.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:515: TEST_auto_repair_bluestore_basic: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:23:10.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2026-02-22T20:23:10.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:23:10.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:23:10.018 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-22T20:23:10.019 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:23:10.019 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:23:10.019 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:23:10.019 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:23:10.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:23:10.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:23:10.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:519: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:23:10.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:23:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:23:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:23:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:23:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:23:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:23:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:23:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:23:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-22T20:23:10.852 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-22T20:23:10.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:23:10.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:10.965 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:10.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:23:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:23:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:23:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:23:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:23:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:23:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:23:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:23:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:23:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:23:10.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:23:10.967 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:23:10.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:23:10.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:23:10.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:23:10.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:23:10.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:23:10.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:23:10.987 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:10.980+0000 7f39538a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:10.989 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:10.984+0000 7f39538a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:10.990 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:10.984+0000 7f39538a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:11.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:23:11.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:23:11.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:23:11.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:23:11.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:23:11.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:11.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:23:11.130 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:23:11.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:11.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:23:11.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:23:11.366 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:11.360+0000 7f39538a3a40 -1 Falling back to public interface 2026-02-22T20:23:12.027 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:12.020+0000 7f39538a3a40 -1 osd.0 20 log_to_monitors true 2026-02-22T20:23:12.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:23:12.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:12.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:23:12.286 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:23:12.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:12.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:23:12.439 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/2470332115,v1:127.0.0.1:6803/2470332115] [v2:127.0.0.1:6804/2470332115,v1:127.0.0.1:6805/2470332115] exists,up fd4cbde9-c19a-402a-b381-fb310894d983 2026-02-22T20:23:12.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:23:12.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:23:12.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:23:12.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:23:12.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:23:12.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:23:12.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:23:12.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:23:12.439 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:23:12.440 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:23:12.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:23:12.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:23:12.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:23:12.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:23:12.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:23:12.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:23:12.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:23:12.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:23:12.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:23:12.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:23:12.708 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:23:12.708 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:23:12.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:23:12.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:12.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:23:12.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-22T20:23:12.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-22T20:23:12.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-22T20:23:12.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:12.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:23:12.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672980 2026-02-22T20:23:12.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672980 2026-02-22T20:23:12.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-42949672980' 2026-02-22T20:23:12.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:12.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:23:12.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509456 2026-02-22T20:23:12.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509456 2026-02-22T20:23:12.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-42949672980 2-64424509456' 2026-02-22T20:23:12.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:12.927 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-22T20:23:12.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:12.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:23:12.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-22T20:23:12.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:12.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-22T20:23:12.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-22T20:23:12.931 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 103079215106 2026-02-22T20:23:12.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:13.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-22T20:23:13.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:23:14.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:23:14.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:14.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-22T20:23:14.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:23:15.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:23:15.265 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:15.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215108 -lt 103079215106 2026-02-22T20:23:15.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:15.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672980 2026-02-22T20:23:15.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:15.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:23:15.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672980 2026-02-22T20:23:15.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:15.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672980 2026-02-22T20:23:15.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2026-02-22T20:23:15.424 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672980 2026-02-22T20:23:15.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:23:15.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672981 -lt 42949672980 2026-02-22T20:23:15.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:15.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509456 2026-02-22T20:23:15.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:15.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:23:15.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509456 2026-02-22T20:23:15.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:15.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509456 2026-02-22T20:23:15.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2026-02-22T20:23:15.594 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509456 2026-02-22T20:23:15.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:23:15.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509457 -lt 64424509456 2026-02-22T20:23:15.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:23:15.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:15.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:15.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:23:15.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:23:15.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:23:15.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:23:15.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:23:15.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:23:15.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:23:15.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:23:16.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:23:16.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:23:16.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:16.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:16.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:23:16.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:23:16.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:23:16.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:520: TEST_auto_repair_bluestore_basic: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2026-02-22T20:23:16.377 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:23:16.377 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:23:16.377 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:23:16.384 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:23:16.384 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2026-02-22T20:23:16.384 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:23:16.391 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:23:16.392 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2026-02-22T20:23:16.392 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:23:16.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_basic: get_pg testpool SOMETHING 2026-02-22T20:23:16.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-22T20:23:16.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:23:16.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2026-02-22T20:23:16.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:23:16.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:522: TEST_auto_repair_bluestore_basic: local pgid=1.0 2026-02-22T20:23:16.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_basic: get_primary testpool SOMETHING 2026-02-22T20:23:16.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:23:16.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:23:16.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:23:16.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:23:16.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:523: TEST_auto_repair_bluestore_basic: local primary=1 2026-02-22T20:23:16.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_basic: get_last_scrub_stamp 1.0 2026-02-22T20:23:16.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:23:16.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:23:16.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:23:16.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:23:16.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:524: TEST_auto_repair_bluestore_basic: local last_scrub_stamp=2026-02-22T20:23:01.099756+0000 2026-02-22T20:23:16.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:527: TEST_auto_repair_bluestore_basic: ceph tell 1.0 schedule-deep-scrub 2026-02-22T20:23:16.945 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:23:16.945 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:23:16.945 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:23:16.945 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T15:35:06.941122+0000" 2026-02-22T20:23:16.945 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:23:16.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:530: TEST_auto_repair_bluestore_basic: wait_for_scrub 1.0 2026-02-22T20:23:01.099756+0000 2026-02-22T20:23:16.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:23:16.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:23:01.099756+0000 2026-02-22T20:23:16.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:23:16.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:23:16.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:23:16.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:23:16.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:23:16.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:23:16.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:23:16.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:23:17.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:23:01.099756+0000 '>' 2026-02-22T20:23:01.099756+0000 2026-02-22T20:23:17.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:23:18.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:23:18.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:23:18.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:23:18.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:23:18.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:23:18.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:23:18.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:23:18.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:23:01.099756+0000 '>' 2026-02-22T20:23:01.099756+0000 2026-02-22T20:23:18.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:23:19.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:23:19.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:23:19.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:23:19.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:23:19.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:23:19.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:23:19.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:23:19.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:23:01.099756+0000 '>' 2026-02-22T20:23:01.099756+0000 2026-02-22T20:23:19.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:23:20.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:23:20.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:23:20.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:23:20.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:23:20.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:23:20.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:23:20.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:23:20.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:23:17.905158+0000 '>' 2026-02-22T20:23:01.099756+0000 2026-02-22T20:23:20.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:23:20.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:531: TEST_auto_repair_bluestore_basic: wait_for_clean 2026-02-22T20:23:20.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:23:20.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:23:20.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:23:20.590 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:23:20.590 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:23:20.590 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:23:20.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:23:20.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:23:20.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:23:20.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:23:20.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:23:20.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:23:20.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:23:20.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:23:20.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:23:20.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:23:20.865 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:23:20.865 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:23:20.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:23:20.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:20.865 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:23:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215115 2026-02-22T20:23:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215115 2026-02-22T20:23:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215115' 2026-02-22T20:23:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:20.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:23:21.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672989 2026-02-22T20:23:21.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672989 2026-02-22T20:23:21.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215115 1-42949672989' 2026-02-22T20:23:21.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:21.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:23:21.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509465 2026-02-22T20:23:21.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509465 2026-02-22T20:23:21.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215115 1-42949672989 2-64424509465' 2026-02-22T20:23:21.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:21.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215115 2026-02-22T20:23:21.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:21.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:23:21.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215115 2026-02-22T20:23:21.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:21.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215115 2026-02-22T20:23:21.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215115' 2026-02-22T20:23:21.084 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 103079215115 2026-02-22T20:23:21.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:21.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215114 -lt 103079215115 2026-02-22T20:23:21.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:23:22.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:23:22.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:22.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215117 -lt 103079215115 2026-02-22T20:23:22.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:22.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672989 2026-02-22T20:23:22.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:22.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:23:22.409 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672989 2026-02-22T20:23:22.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:22.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672989 2026-02-22T20:23:22.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2026-02-22T20:23:22.411 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672989 2026-02-22T20:23:22.412 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:23:22.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672990 -lt 42949672989 2026-02-22T20:23:22.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:22.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509465 2026-02-22T20:23:22.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:22.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:23:22.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509465 2026-02-22T20:23:22.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:22.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509465 2026-02-22T20:23:22.582 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509465 2026-02-22T20:23:22.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2026-02-22T20:23:22.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:23:22.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509466 -lt 64424509465 2026-02-22T20:23:22.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:23:22.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:22.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:22.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:23:22.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:23:22.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:23:22.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:23:22.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:23:22.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:23:22.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:23:22.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:23:23.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:23:23.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:23:23.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:23.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:23.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:23:23.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:23:23.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:23:23.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:532: TEST_auto_repair_bluestore_basic: ceph pg dump pgs 2026-02-22T20:23:23.429 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:23:23.430 INFO:tasks.workunit.client.0.trial150.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-22T20:23:17.907293+0000 20'1 25:68 [1,0] 1 [1,0] 1 20'1 2026-02-22T20:23:17.905158+0000 20'1 2026-02-22T20:23:17.905158+0000 0 1 periodic scrub scheduled @ 2026-02-23T20:23:17.905158+0000 1 0 2026-02-22T20:23:23.430 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:23:23.430 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:23:23.430 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:23:23.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2026-02-22T20:23:23.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:23:23.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:23:23.441 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-22T20:23:23.441 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:23:23.441 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:23:23.441 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:23:23.441 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:23:23.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:23:23.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:23:23.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:535: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:23:23.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:23:23.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:23:23.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:23:23.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:23:24.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:23:24.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:23:24.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:23:24.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:23:24.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:23:24.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:23:24.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:23:24.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-22T20:23:24.156 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:23:24.156 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:23:24.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:23:24.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:23:24.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:23:24.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:23:24.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:23:24.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:23:24.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:23:24.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:23:24.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:23:24.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:23:24.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:23:24.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:23:24.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:23:24.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:23:24.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:23:24.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:23:24.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:23:24.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:23:24.264 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:23:24.264 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:24.264 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:24.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:24.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:24.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:23:24.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:23:24.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:23:24.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:23:24.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:23:24.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:23:24.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:23:24.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:23:24.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:23:24.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:23:24.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:23:24.267 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:23:24.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:23:24.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:23:24.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:23:24.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:23:24.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:23:24.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:23:24.289 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:24.283+0000 7ffb855f0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:24.290 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:24.283+0000 7ffb855f0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:24.291 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:24.283+0000 7ffb855f0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:24.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:23:24.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:23:24.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:23:24.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:23:24.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:23:24.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:24.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:23:24.424 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:23:24.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:24.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:23:24.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:23:24.698 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:24.691+0000 7ffb855f0a40 -1 Falling back to public interface 2026-02-22T20:23:25.339 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:25.331+0000 7ffb855f0a40 -1 osd.0 25 log_to_monitors true 2026-02-22T20:23:25.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:23:25.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:25.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:23:25.584 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:23:25.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:25.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:23:25.743 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/1598716680,v1:127.0.0.1:6803/1598716680] [v2:127.0.0.1:6804/1598716680,v1:127.0.0.1:6805/1598716680] exists,up fd4cbde9-c19a-402a-b381-fb310894d983 2026-02-22T20:23:25.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:23:25.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:23:25.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:23:25.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:23:25.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:23:25.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:23:25.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:23:25.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:23:25.745 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:23:25.745 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:23:25.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:23:25.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:23:25.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:23:25.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:23:25.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:23:25.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:23:25.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:23:25.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:23:25.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:23:26.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:23:26.004 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:23:26.004 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:23:26.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:23:26.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:26.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:23:26.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-22T20:23:26.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-22T20:23:26.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586' 2026-02-22T20:23:26.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:26.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:23:26.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672995 2026-02-22T20:23:26.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672995 2026-02-22T20:23:26.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-42949672995' 2026-02-22T20:23:26.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:26.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:23:26.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509471 2026-02-22T20:23:26.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509471 2026-02-22T20:23:26.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-42949672995 2-64424509471' 2026-02-22T20:23:26.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:26.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-124554051586 2026-02-22T20:23:26.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:26.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:23:26.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-124554051586 2026-02-22T20:23:26.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:26.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-22T20:23:26.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2026-02-22T20:23:26.223 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 124554051586 2026-02-22T20:23:26.224 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:26.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051586 -lt 124554051586 2026-02-22T20:23:26.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:26.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672995 2026-02-22T20:23:26.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:26.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:23:26.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672995 2026-02-22T20:23:26.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:26.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672995 2026-02-22T20:23:26.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2026-02-22T20:23:26.382 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672995 2026-02-22T20:23:26.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:23:26.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672995 -lt 42949672995 2026-02-22T20:23:26.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:26.548 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509471 2026-02-22T20:23:26.548 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:26.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:23:26.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509471 2026-02-22T20:23:26.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:26.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509471 2026-02-22T20:23:26.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2026-02-22T20:23:26.552 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509471 2026-02-22T20:23:26.552 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:23:26.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509471 -lt 64424509471 2026-02-22T20:23:26.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:23:26.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:26.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:26.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:23:26.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:23:26.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:23:26.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:23:26.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:23:26.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:23:26.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:23:26.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:23:27.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:23:27.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:23:27.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:27.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:27.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:23:27.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:23:27.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:23:27.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:536: TEST_auto_repair_bluestore_basic: get_not_primary testpool SOMETHING 2026-02-22T20:23:27.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:23:27.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:23:27.270 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-22T20:23:27.270 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:23:27.270 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:23:27.271 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:23:27.271 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:23:27.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:23:27.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:23:27.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:23:27.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:536: TEST_auto_repair_bluestore_basic: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-22T20:23:27.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:23:27.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:23:27.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:23:27.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:23:27.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-22T20:23:27.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:23:27.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:23:27.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:23:27.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:23:27.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:23:27.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:23:27.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:23:27.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:23:27.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:23:27.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:23:27.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:23:27.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-22T20:23:27.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:23:27.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:23:27.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:23:27.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:23:27.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:23:27.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-22T20:23:28.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:23:28.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:23:28.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:23:28.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:23:28.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:23:28.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:23:28.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:23:28.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:23:28.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:23:28.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:23:28.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:23:28.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:23:28.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:23:28.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:23:28.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:23:28.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:23:28.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:23:28.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:23:28.085 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:23:28.085 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:28.085 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:28.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:28.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:28.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:23:28.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:23:28.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:23:28.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:23:28.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:23:28.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:23:28.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:23:28.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:23:28.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:23:28.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:23:28.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:23:28.087 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:23:28.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:23:28.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:23:28.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:23:28.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:23:28.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:23:28.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:23:28.101 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:28.095+0000 7fe6d5f67a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:28.102 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:28.095+0000 7fe6d5f67a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:28.103 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:28.095+0000 7fe6d5f67a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:28.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:23:28.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:23:28.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:23:28.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:23:28.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:23:28.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:28.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:23:28.248 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:23:28.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:28.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:23:28.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:23:28.478 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:28.471+0000 7fe6d5f67a40 -1 Falling back to public interface 2026-02-22T20:23:29.042 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:29.035+0000 7fe6d5f67a40 -1 osd.0 30 log_to_monitors true 2026-02-22T20:23:29.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:23:29.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:29.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:23:29.408 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:23:29.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:29.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:23:29.564 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6802/361289460,v1:127.0.0.1:6803/361289460] [v2:127.0.0.1:6804/361289460,v1:127.0.0.1:6805/361289460] exists,up fd4cbde9-c19a-402a-b381-fb310894d983 2026-02-22T20:23:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:23:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:23:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:23:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:23:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:23:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:23:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:23:29.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:23:29.565 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:23:29.565 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:23:29.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:23:29.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:23:29.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:23:29.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:23:29.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:23:29.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:23:29.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:23:29.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:23:29.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:23:29.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:23:29.816 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:23:29.817 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:23:29.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:23:29.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:29.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:23:29.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888066 2026-02-22T20:23:29.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888066 2026-02-22T20:23:29.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066' 2026-02-22T20:23:29.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:29.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:23:29.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673000 2026-02-22T20:23:29.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673000 2026-02-22T20:23:29.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066 1-42949673000' 2026-02-22T20:23:29.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:29.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:23:30.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509476 2026-02-22T20:23:30.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509476 2026-02-22T20:23:30.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066 1-42949673000 2-64424509476' 2026-02-22T20:23:30.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:30.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-146028888066 2026-02-22T20:23:30.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:30.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:23:30.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-146028888066 2026-02-22T20:23:30.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:30.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888066 2026-02-22T20:23:30.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2026-02-22T20:23:30.041 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 146028888066 2026-02-22T20:23:30.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:30.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051588 -lt 146028888066 2026-02-22T20:23:30.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:23:31.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:23:31.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:31.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888067 -lt 146028888066 2026-02-22T20:23:31.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:31.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673000 2026-02-22T20:23:31.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:31.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:23:31.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673000 2026-02-22T20:23:31.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:31.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673000 2026-02-22T20:23:31.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2026-02-22T20:23:31.369 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949673000 2026-02-22T20:23:31.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:23:31.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673000 -lt 42949673000 2026-02-22T20:23:31.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:31.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509476 2026-02-22T20:23:31.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:31.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:23:31.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509476 2026-02-22T20:23:31.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509476 2026-02-22T20:23:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2026-02-22T20:23:31.533 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509476 2026-02-22T20:23:31.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:23:31.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509476 -lt 64424509476 2026-02-22T20:23:31.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:23:31.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:31.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:31.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:23:31.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:23:31.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:23:31.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:23:31.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:23:31.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:23:31.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:23:31.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:23:32.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:23:32.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:23:32.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:32.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:32.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:23:32.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:23:32.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:23:32.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:537: TEST_auto_repair_bluestore_basic: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:23:32.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:538: TEST_auto_repair_bluestore_basic: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-22T20:23:32.237 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:23:17.904+0000 7fa273898640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-22T20:23:32.237 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:23:17.904+0000 7fa273898640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-22T20:23:32.237 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:23:17.904+0000 7fa273898640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2026-02-22T20:23:32.237 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:23:17.904+0000 7fa273898640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2026-02-22T20:23:32.237 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:23:17.904+0000 7fa273898640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-22T20:23:32.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:23:32.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:23:32.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:23:32.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:23:32.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:23:32.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:23:32.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:23:32.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:23:32.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:23:32.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:23:32.348 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:23:32.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:23:32.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:23:32.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:23:32.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:23:32.352 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:23:32.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:23:32.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:23:32.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:23:32.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:23:32.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:23:32.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:23:32.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:23:32.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:23:32.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:32.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:23:32.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:23:32.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:23:32.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:23:32.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:23:32.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:23:32.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:23:32.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:23:32.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:23:32.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:23:32.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:23:32.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:23:32.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:23:32.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:23:32.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:23:32.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:23:32.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:23:32.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:23:32.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:23:32.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:23:32.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:23:32.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:23:32.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:23:32.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:23:32.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:23:32.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:23:32.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:23:32.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:23:32.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:23:32.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:23:32.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:32.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:32.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:23:32.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:23:32.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:23:32.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:23:32.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:23:32.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:32.387 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:32.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:23:32.388 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:23:32.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:23:32.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:23:32.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:23:32.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed td/osd-scrub-repair 2026-02-22T20:23:32.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:598: TEST_auto_repair_bluestore_failed: local dir=td/osd-scrub-repair 2026-02-22T20:23:32.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:599: TEST_auto_repair_bluestore_failed: local poolname=testpool 2026-02-22T20:23:32.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:602: TEST_auto_repair_bluestore_failed: run_mon td/osd-scrub-repair a 2026-02-22T20:23:32.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:23:32.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:23:32.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:23:32.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:23:32.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:23:32.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:23:32.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:23:32.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:23:32.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:23:32.411 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:23:32.411 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:32.411 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:32.412 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:32.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:23:32.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:23:32.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:23:32.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:23:32.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:23:32.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:23:32.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:23:32.442 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:23:32.442 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:23:32.442 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:23:32.443 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:23:32.443 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:32.443 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:32.443 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:23:32.444 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:23:32.444 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:23:32.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:23:32.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:23:32.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:23:32.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:23:32.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:23:32.502 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:23:32.502 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:23:32.502 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:23:32.502 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:23:32.502 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:32.502 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:32.502 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:23:32.503 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:23:32.503 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:23:32.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:603: TEST_auto_repair_bluestore_failed: run_mgr td/osd-scrub-repair x 2026-02-22T20:23:32.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:23:32.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:23:32.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:23:32.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:23:32.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:23:32.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:23:32.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:23:32.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:23:32.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:23:32.654 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:23:32.654 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:32.654 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:32.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:32.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:23:32.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:23:32.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:606: TEST_auto_repair_bluestore_failed: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:23:32.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:607: TEST_auto_repair_bluestore_failed: seq 0 2 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:607: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:23:32.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:23:32.677 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:23:32.677 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:32.677 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:32.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:32.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:32.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:23:32.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:23:32.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:23:32.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:23:32.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:23:32.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:23:32.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:23:32.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:23:32.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:23:32.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:23:32.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:23:32.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:23:32.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:23:32.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4e867cbb-1c13-4b64-9a4d-9c050ca0c6b6 2026-02-22T20:23:32.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4e867cbb-1c13-4b64-9a4d-9c050ca0c6b6' 2026-02-22T20:23:32.679 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 4e867cbb-1c13-4b64-9a4d-9c050ca0c6b6 2026-02-22T20:23:32.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:23:32.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDEZZtpWoPKKBAA0ZM3hYpFFCFcFXSbgKnqtQ== 2026-02-22T20:23:32.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDEZZtpWoPKKBAA0ZM3hYpFFCFcFXSbgKnqtQ=="}' 2026-02-22T20:23:32.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4e867cbb-1c13-4b64-9a4d-9c050ca0c6b6 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:23:32.778 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:23:32.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:23:32.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDEZZtpWoPKKBAA0ZM3hYpFFCFcFXSbgKnqtQ== --osd-uuid 4e867cbb-1c13-4b64-9a4d-9c050ca0c6b6 2026-02-22T20:23:32.804 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:32.795+0000 7fe31ad86a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:32.805 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:32.799+0000 7fe31ad86a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:32.806 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:32.799+0000 7fe31ad86a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:32.806 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:32.799+0000 7fe31ad86a40 -1 bdev(0x55cb35394c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:23:32.807 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:32.799+0000 7fe31ad86a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:23:33.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:23:33.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:23:33.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:23:33.537 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:23:33.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:23:33.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:23:33.747 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:23:33.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:23:33.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:23:33.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:23:33.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:23:33.772 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:33.767+0000 7ff5eb3cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:33.774 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:33.767+0000 7ff5eb3cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:33.775 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:33.767+0000 7ff5eb3cda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:33.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:23:33.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:23:33.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:23:33.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:23:33.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:23:33.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:33.909 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:23:33.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:23:33.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:33.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:23:34.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:23:34.158 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:34.151+0000 7ff5eb3cda40 -1 Falling back to public interface 2026-02-22T20:23:34.726 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:34.719+0000 7ff5eb3cda40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:23:35.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:23:35.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:35.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:23:35.071 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:23:35.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:23:35.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:35.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:23:36.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:23:36.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:36.229 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:23:36.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:23:36.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:36.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:23:36.388 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1406028671,v1:127.0.0.1:6803/1406028671] [v2:127.0.0.1:6804/1406028671,v1:127.0.0.1:6805/1406028671] exists,up 4e867cbb-1c13-4b64-9a4d-9c050ca0c6b6 2026-02-22T20:23:36.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:23:36.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:23:36.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:23:36.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:607: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2026-02-22T20:23:36.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:23:36.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:23:36.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:23:36.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:23:36.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:23:36.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:23:36.393 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:23:36.394 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:36.394 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:36.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:36.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:36.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:23:36.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:23:36.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:23:36.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:23:36.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:23:36.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:23:36.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:23:36.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:23:36.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:23:36.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:23:36.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:23:36.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:23:36.395 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:23:36.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=85c41e2e-5d63-4607-bded-8b0c5ae85a1c 2026-02-22T20:23:36.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 85c41e2e-5d63-4607-bded-8b0c5ae85a1c' 2026-02-22T20:23:36.396 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 85c41e2e-5d63-4607-bded-8b0c5ae85a1c 2026-02-22T20:23:36.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:23:36.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDIZZtphSDmFxAADdlwVE7iKIqANyuIV7RbVg== 2026-02-22T20:23:36.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDIZZtphSDmFxAADdlwVE7iKIqANyuIV7RbVg=="}' 2026-02-22T20:23:36.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 85c41e2e-5d63-4607-bded-8b0c5ae85a1c -i td/osd-scrub-repair/1/new.json 2026-02-22T20:23:36.570 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:23:36.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:23:36.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDIZZtphSDmFxAADdlwVE7iKIqANyuIV7RbVg== --osd-uuid 85c41e2e-5d63-4607-bded-8b0c5ae85a1c 2026-02-22T20:23:36.598 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:36.591+0000 7f864983ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:36.599 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:36.591+0000 7f864983ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:36.600 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:36.595+0000 7f864983ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:36.601 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:36.595+0000 7f864983ea40 -1 bdev(0x55e69d6a4c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:23:36.601 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:36.595+0000 7f864983ea40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:23:37.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:23:37.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:23:37.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:23:37.221 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:23:37.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:23:37.424 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:23:37.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:23:37.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:23:37.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:23:37.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:23:37.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:23:37.451 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:37.443+0000 7f7d10e0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:37.453 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:37.447+0000 7f7d10e0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:37.454 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:37.447+0000 7f7d10e0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:37.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:23:37.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:23:37.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:23:37.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:23:37.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:23:37.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:37.582 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:23:37.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:23:37.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:37.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:23:37.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:23:37.806 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:37.799+0000 7f7d10e0ea40 -1 Falling back to public interface 2026-02-22T20:23:38.412 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:38.403+0000 7f7d10e0ea40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:23:38.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:23:38.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:38.744 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:23:38.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:23:38.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:38.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:23:38.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:23:39.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:23:39.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:39.904 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:23:39.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:23:39.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:39.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:23:40.063 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2795725817,v1:127.0.0.1:6811/2795725817] [v2:127.0.0.1:6812/2795725817,v1:127.0.0.1:6813/2795725817] exists,up 85c41e2e-5d63-4607-bded-8b0c5ae85a1c 2026-02-22T20:23:40.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:23:40.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:23:40.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:23:40.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:607: TEST_auto_repair_bluestore_failed: for id in $(seq 0 2) 2026-02-22T20:23:40.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:608: TEST_auto_repair_bluestore_failed: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:23:40.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:23:40.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:23:40.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:23:40.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:23:40.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:23:40.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:23:40.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:23:40.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:23:40.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:23:40.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=125a87f3-5548-4288-8e16-70586f73da33 2026-02-22T20:23:40.071 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 125a87f3-5548-4288-8e16-70586f73da33 2026-02-22T20:23:40.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 125a87f3-5548-4288-8e16-70586f73da33' 2026-02-22T20:23:40.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:23:40.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDMZZtp909vBBAAd1Z40KlmVwk3WoGJVTGgaw== 2026-02-22T20:23:40.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDMZZtp909vBBAAd1Z40KlmVwk3WoGJVTGgaw=="}' 2026-02-22T20:23:40.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 125a87f3-5548-4288-8e16-70586f73da33 -i td/osd-scrub-repair/2/new.json 2026-02-22T20:23:40.247 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:23:40.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:23:40.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDMZZtp909vBBAAd1Z40KlmVwk3WoGJVTGgaw== --osd-uuid 125a87f3-5548-4288-8e16-70586f73da33 2026-02-22T20:23:40.274 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:40.267+0000 7f06df83ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:40.276 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:40.267+0000 7f06df83ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:40.277 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:40.271+0000 7f06df83ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:40.278 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:40.271+0000 7f06df83ca40 -1 bdev(0x555bc10fac00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:23:40.278 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:40.271+0000 7f06df83ca40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:23:40.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:23:40.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:23:40.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:23:40.913 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:23:40.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:23:41.105 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:23:41.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:23:41.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:23:41.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:23:41.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:23:41.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:23:41.131 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:41.123+0000 7f6e802c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:41.132 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:41.127+0000 7f6e802c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:41.133 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:41.127+0000 7f6e802c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:41.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:23:41.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:23:41.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:23:41.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:23:41.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:23:41.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:41.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:23:41.260 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:23:41.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:41.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:23:41.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:23:41.526 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:41.519+0000 7f6e802c5a40 -1 Falling back to public interface 2026-02-22T20:23:42.148 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:42.139+0000 7f6e802c5a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:23:42.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:23:42.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:42.426 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:23:42.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:23:42.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:42.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:23:42.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:23:43.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:23:43.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:43.587 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:23:43.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:23:43.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:43.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:23:43.746 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1284242446,v1:127.0.0.1:6819/1284242446] [v2:127.0.0.1:6820/1284242446,v1:127.0.0.1:6821/1284242446] exists,up 125a87f3-5548-4288-8e16-70586f73da33 2026-02-22T20:23:43.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:23:43.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:23:43.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:23:43.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:611: TEST_auto_repair_bluestore_failed: create_pool testpool 1 1 2026-02-22T20:23:43.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-22T20:23:43.930 INFO:tasks.workunit.client.0.trial150.stderr:pool 'testpool' created 2026-02-22T20:23:43.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:23:44.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:612: TEST_auto_repair_bluestore_failed: ceph osd pool set testpool size 2 2026-02-22T20:23:45.144 INFO:tasks.workunit.client.0.trial150.stderr:set pool 1 size to 2 2026-02-22T20:23:45.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:613: TEST_auto_repair_bluestore_failed: wait_for_clean 2026-02-22T20:23:45.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:23:45.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:23:45.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:23:45.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:23:45.158 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:23:45.158 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:23:45.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:23:45.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:23:45.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:23:45.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:23:45.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:23:45.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:23:45.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:23:45.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:23:45.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:23:45.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:23:45.436 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:23:45.437 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:23:45.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:23:45.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:45.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:23:45.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-22T20:23:45.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-22T20:23:45.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T20:23:45.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:45.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:23:45.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-22T20:23:45.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-22T20:23:45.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-22T20:23:45.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:45.590 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:23:45.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574851 2026-02-22T20:23:45.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574851 2026-02-22T20:23:45.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-55834574851' 2026-02-22T20:23:45.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:45.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-22T20:23:45.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:45.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:23:45.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-22T20:23:45.666 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:45.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-22T20:23:45.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T20:23:45.667 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836484 2026-02-22T20:23:45.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:45.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:23:45.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:23:46.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:23:46.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:46.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:23:46.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:23:47.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:23:47.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:48.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836484 2026-02-22T20:23:48.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:48.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-22T20:23:48.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:48.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:23:48.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-22T20:23:48.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:48.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-22T20:23:48.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-22T20:23:48.158 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705667 2026-02-22T20:23:48.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:23:48.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-22T20:23:48.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:48.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574851 2026-02-22T20:23:48.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:48.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:23:48.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574851 2026-02-22T20:23:48.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:48.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574851 2026-02-22T20:23:48.324 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574851 2026-02-22T20:23:48.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574851' 2026-02-22T20:23:48.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:23:48.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574851 -lt 55834574851 2026-02-22T20:23:48.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:23:48.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:48.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:48.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:23:48.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:23:48.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:23:48.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:23:48.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:23:48.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:23:48.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:23:48.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:23:48.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:23:48.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:23:48.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:48.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:49.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:23:49.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-22T20:23:49.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:23:49.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-22T20:23:49.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:23:49.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:23:49.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:23:49.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:23:49.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:23:49.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:23:49.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:23:49.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:23:49.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:23:49.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:23:49.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:23:49.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:23:49.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:49.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:49.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:23:49.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:23:49.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:23:49.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:23:49.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:23:49.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:23:49.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:23:49.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:23:49.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:23:49.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:23:49.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:23:49.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:23:49.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:23:49.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:23:49.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:23:49.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:23:49.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:23:49.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:23:49.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:23:49.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:23:49.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:23:49.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:23:50.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:23:50.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:23:50.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:50.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:50.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:23:50.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:23:50.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:23:50.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:23:50.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:23:50.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:23:50.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:23:50.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:23:50.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:23:50.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:23:50.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:23:50.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-22T20:23:50.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:23:50.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-22T20:23:50.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:23:50.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:23:50.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:23:50.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:23:50.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:23:50.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:23:50.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:23:50.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:23:50.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:23:50.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:23:50.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:50.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:51.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:23:51.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:23:51.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:23:51.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:23:51.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:23:51.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:23:51.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:23:51.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:23:51.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:23:51.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:23:51.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:23:51.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-22T20:23:51.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:23:51.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-22T20:23:52.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:23:52.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:23:52.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:23:52.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:23:52.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:23:52.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:23:52.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:23:52.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:23:52.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:23:52.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:23:52.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:52.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:52.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:23:52.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:23:52.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:23:52.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:616: TEST_auto_repair_bluestore_failed: local payload=ABCDEF 2026-02-22T20:23:52.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:617: TEST_auto_repair_bluestore_failed: echo ABCDEF 2026-02-22T20:23:52.508 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: seq 1 10 2026-02-22T20:23:52.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-22T20:23:52.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:23:52.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-22T20:23:52.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:23:52.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-22T20:23:52.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:23:52.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-22T20:23:52.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:23:52.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-22T20:23:52.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:23:52.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-22T20:23:52.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:23:52.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-22T20:23:52.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:23:52.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-22T20:23:52.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:23:52.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-22T20:23:52.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:23:52.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:618: TEST_auto_repair_bluestore_failed: for i in $(seq 1 10) 2026-02-22T20:23:52.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:620: TEST_auto_repair_bluestore_failed: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:23:52.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2026-02-22T20:23:52.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:23:52.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:23:52.741 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-22T20:23:52.741 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:23:52.741 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:23:52.741 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:23:52.742 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:23:52.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:23:52.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:23:52.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:23:53.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:625: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2026-02-22T20:23:53.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:23:53.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:23:53.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:23:53.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:23:53.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2026-02-22T20:23:53.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:23:53.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:23:53.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:23:53.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:23:53.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:23:53.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:23:53.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:23:53.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:23:53.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:23:53.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:23:53.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:23:53.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2026-02-22T20:23:53.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:23:53.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:23:53.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:23:53.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:23:53.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:23:53.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2026-02-22T20:23:53.548 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:23:53.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:23:53.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:23:53.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:23:53.638 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:23:53.638 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:53.638 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:53.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:53.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:53.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:23:53.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:23:53.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:23:53.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:23:53.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:23:53.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:23:53.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:23:53.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:23:53.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:23:53.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:23:53.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:23:53.640 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:23:53.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:23:53.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:23:53.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:23:53.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:23:53.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:23:53.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:23:53.661 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:53.655+0000 7f6d5a2a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:53.662 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:53.655+0000 7f6d5a2a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:53.663 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:53.655+0000 7f6d5a2a0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:23:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:23:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:23:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:23:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:23:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:23:53.804 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:23:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:23:53.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:23:54.014 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:54.007+0000 7f6d5a2a0a40 -1 Falling back to public interface 2026-02-22T20:23:54.664 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:54.655+0000 7f6d5a2a0a40 -1 osd.0 18 log_to_monitors true 2026-02-22T20:23:54.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:23:54.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:23:54.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:23:54.967 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:23:54.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:23:54.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:23:55.130 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6802/893294296,v1:127.0.0.1:6803/893294296] [v2:127.0.0.1:6804/893294296,v1:127.0.0.1:6805/893294296] exists,up 4e867cbb-1c13-4b64-9a4d-9c050ca0c6b6 2026-02-22T20:23:55.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:23:55.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:23:55.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:23:55.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:23:55.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:23:55.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:23:55.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:23:55.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:23:55.131 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:23:55.131 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:23:55.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:23:55.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:23:55.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:23:55.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:23:55.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:23:55.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:23:55.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:23:55.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:23:55.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:23:55.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:23:55.398 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:23:55.398 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:23:55.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:23:55.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:55.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:23:55.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280514 2026-02-22T20:23:55.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280514 2026-02-22T20:23:55.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514' 2026-02-22T20:23:55.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:55.473 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:23:55.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705670 2026-02-22T20:23:55.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705670 2026-02-22T20:23:55.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514 1-38654705670' 2026-02-22T20:23:55.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:23:55.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:23:55.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574854 2026-02-22T20:23:55.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574854 2026-02-22T20:23:55.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514 1-38654705670 2-55834574854' 2026-02-22T20:23:55.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:55.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280514 2026-02-22T20:23:55.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:55.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:23:55.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280514 2026-02-22T20:23:55.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:55.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280514 2026-02-22T20:23:55.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280514' 2026-02-22T20:23:55.628 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 94489280514 2026-02-22T20:23:55.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:55.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 94489280514 2026-02-22T20:23:55.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:23:56.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:23:56.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:56.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 94489280514 2026-02-22T20:23:56.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:23:57.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:23:57.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:23:58.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280514 -lt 94489280514 2026-02-22T20:23:58.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:58.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705670 2026-02-22T20:23:58.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:58.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:23:58.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705670 2026-02-22T20:23:58.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:58.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705670 2026-02-22T20:23:58.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2026-02-22T20:23:58.115 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 38654705670 2026-02-22T20:23:58.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:23:58.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705670 -lt 38654705670 2026-02-22T20:23:58.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:23:58.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574854 2026-02-22T20:23:58.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:23:58.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:23:58.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574854 2026-02-22T20:23:58.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:23:58.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574854 2026-02-22T20:23:58.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574854' 2026-02-22T20:23:58.278 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 55834574854 2026-02-22T20:23:58.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:23:58.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574854 -lt 55834574854 2026-02-22T20:23:58.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:23:58.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:58.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:58.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:23:58.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:23:58.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:23:58.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:23:58.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:23:58.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:23:58.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:23:58.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:23:58.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:23:58.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:23:58.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:23:58.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:23:58.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:23:58.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:23:58.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:23:58.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed: get_not_primary testpool SOMETHING 2026-02-22T20:23:58.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:23:58.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:23:58.996 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-22T20:23:58.996 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:23:58.996 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:23:58.996 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:23:58.996 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:23:59.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:23:59.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:23:59.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:23:59.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:627: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2026-02-22T20:23:59.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:23:59.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:23:59.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:23:59.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:23:59.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2026-02-22T20:23:59.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:23:59.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:23:59.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:23:59.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:23:59.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:23:59.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:23:59.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:23:59.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:23:59.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:23:59.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:23:59.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:23:59.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2026-02-22T20:23:59.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:23:59.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:23:59.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:23:59.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:23:59.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:23:59.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2026-02-22T20:23:59.796 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:104778fc:::obj2:head# 2026-02-22T20:23:59.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:23:59.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:23:59.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:23:59.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:23:59.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:23:59.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:23:59.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:23:59.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:23:59.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:23:59.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:23:59.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:23:59.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:23:59.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:23:59.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:23:59.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:23:59.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:23:59.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:23:59.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:23:59.894 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:23:59.894 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:23:59.894 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:23:59.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:59.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:23:59.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:23:59.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:23:59.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:23:59.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:23:59.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:23:59.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:23:59.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:23:59.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:23:59.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:23:59.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:23:59.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:23:59.896 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:23:59.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:23:59.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:23:59.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:23:59.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:23:59.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:23:59.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:23:59.911 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:59.903+0000 7f6057583a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:59.912 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:59.903+0000 7f6057583a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:23:59.913 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:23:59.907+0000 7f6057583a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:00.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:24:00.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:24:00.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:24:00.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:24:00.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:24:00.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:00.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:24:00.056 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:24:00.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:00.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:24:00.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:24:00.350 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:00.343+0000 7f6057583a40 -1 Falling back to public interface 2026-02-22T20:24:00.998 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:00.991+0000 7f6057583a40 -1 osd.0 24 log_to_monitors true 2026-02-22T20:24:01.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:24:01.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:01.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:24:01.216 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:24:01.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:01.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:24:01.371 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 28 up_thru 0 down_at 25 last_clean_interval [22,24) [v2:127.0.0.1:6802/4010042603,v1:127.0.0.1:6803/4010042603] [v2:127.0.0.1:6804/4010042603,v1:127.0.0.1:6805/4010042603] exists,up 4e867cbb-1c13-4b64-9a4d-9c050ca0c6b6 2026-02-22T20:24:01.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:24:01.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:24:01.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:24:01.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:24:01.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:24:01.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:24:01.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:24:01.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:24:01.372 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:24:01.372 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:24:01.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:24:01.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:24:01.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:24:01.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:24:01.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:24:01.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:24:01.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:24:01.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:24:01.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:24:01.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:24:01.647 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:24:01.647 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:24:01.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:24:01.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:01.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:24:01.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084290 2026-02-22T20:24:01.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084290 2026-02-22T20:24:01.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290' 2026-02-22T20:24:01.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:01.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:24:01.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-22T20:24:01.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-22T20:24:01.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290 1-38654705672' 2026-02-22T20:24:01.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:01.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:24:01.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574856 2026-02-22T20:24:01.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574856 2026-02-22T20:24:01.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290 1-38654705672 2-55834574856' 2026-02-22T20:24:01.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:01.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-120259084290 2026-02-22T20:24:01.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:01.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:24:01.865 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-120259084290 2026-02-22T20:24:01.866 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:01.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084290 2026-02-22T20:24:01.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2026-02-22T20:24:01.867 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 120259084290 2026-02-22T20:24:01.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:02.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 120259084290 2026-02-22T20:24:02.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:24:03.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:24:03.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:03.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 120259084290 2026-02-22T20:24:03.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:24:04.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:24:04.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:04.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084290 -lt 120259084290 2026-02-22T20:24:04.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:04.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-22T20:24:04.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:04.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:24:04.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-22T20:24:04.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:04.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-22T20:24:04.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-22T20:24:04.361 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 38654705672 2026-02-22T20:24:04.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:24:04.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-22T20:24:04.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:04.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574856 2026-02-22T20:24:04.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:04.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:24:04.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574856 2026-02-22T20:24:04.520 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:04.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574856 2026-02-22T20:24:04.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574856' 2026-02-22T20:24:04.521 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 55834574856 2026-02-22T20:24:04.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:24:04.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574856 -lt 55834574856 2026-02-22T20:24:04.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:24:04.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:04.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:04.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:24:04.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:24:04.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:24:04.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:24:04.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:24:04.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:24:04.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:24:04.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:24:05.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:24:05.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:24:05.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:05.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:05.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:24:05.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:24:05.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:24:05.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:628: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2026-02-22T20:24:05.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:24:05.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:24:05.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:24:05.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:24:05.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:628: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-22T20:24:05.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:24:05.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:24:05.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:24:05.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:24:05.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-22T20:24:05.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:24:05.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:24:05.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:24:05.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:24:05.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:24:05.391 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:24:05.391 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:24:05.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:24:05.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:24:05.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:24:05.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:24:05.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-22T20:24:05.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:24:05.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:24:05.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:24:05.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:24:05.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:24:05.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:24:06.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:24:06.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:24:06.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:24:06.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:24:06.014 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:24:06.014 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:06.014 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:06.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:24:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:24:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:24:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:24:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:24:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:24:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:24:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:24:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:24:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:24:06.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:24:06.016 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:24:06.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:24:06.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:24:06.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:24:06.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:24:06.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:24:06.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:24:06.037 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:06.031+0000 7f489930fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:06.038 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:06.031+0000 7f489930fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:06.039 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:06.031+0000 7f489930fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:06.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:24:06.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:24:06.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:24:06.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:24:06.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:24:06.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:06.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:24:06.175 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:24:06.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:06.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:24:06.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:24:06.414 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:06.407+0000 7f489930fa40 -1 Falling back to public interface 2026-02-22T20:24:07.016 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:07.007+0000 7f489930fa40 -1 osd.1 30 log_to_monitors true 2026-02-22T20:24:07.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:24:07.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:07.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:24:07.338 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:24:07.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:07.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:24:07.494 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 34 up_thru 34 down_at 30 last_clean_interval [9,30) [v2:127.0.0.1:6810/2427369384,v1:127.0.0.1:6811/2427369384] [v2:127.0.0.1:6812/2427369384,v1:127.0.0.1:6813/2427369384] exists,up 85c41e2e-5d63-4607-bded-8b0c5ae85a1c 2026-02-22T20:24:07.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:24:07.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:24:07.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:24:07.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:24:07.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:24:07.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:24:07.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:24:07.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:24:07.495 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:24:07.495 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:24:07.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:24:07.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:24:07.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:24:07.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:24:07.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:24:07.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:24:07.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:24:07.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:24:07.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:24:07.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:24:07.769 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:24:07.769 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:24:07.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:24:07.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:07.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:24:07.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084292 2026-02-22T20:24:07.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084292 2026-02-22T20:24:07.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084292' 2026-02-22T20:24:07.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:07.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:24:07.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888066 2026-02-22T20:24:07.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888066 2026-02-22T20:24:07.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084292 1-146028888066' 2026-02-22T20:24:07.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:07.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:24:07.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574858 2026-02-22T20:24:07.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574858 2026-02-22T20:24:07.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084292 1-146028888066 2-55834574858' 2026-02-22T20:24:07.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:07.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-120259084292 2026-02-22T20:24:07.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:07.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:24:07.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-120259084292 2026-02-22T20:24:07.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:08.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084292 2026-02-22T20:24:08.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 120259084292' 2026-02-22T20:24:08.000 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 120259084292 2026-02-22T20:24:08.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:08.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084291 -lt 120259084292 2026-02-22T20:24:08.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:24:09.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:24:09.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:09.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084292 -lt 120259084292 2026-02-22T20:24:09.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:09.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-146028888066 2026-02-22T20:24:09.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:09.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:24:09.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-146028888066 2026-02-22T20:24:09.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:09.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888066 2026-02-22T20:24:09.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 146028888066' 2026-02-22T20:24:09.325 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 146028888066 2026-02-22T20:24:09.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:24:09.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888066 -lt 146028888066 2026-02-22T20:24:09.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:09.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574858 2026-02-22T20:24:09.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:09.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:24:09.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574858 2026-02-22T20:24:09.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:09.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574858 2026-02-22T20:24:09.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574858' 2026-02-22T20:24:09.488 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 55834574858 2026-02-22T20:24:09.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:24:09.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574859 -lt 55834574858 2026-02-22T20:24:09.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:24:09.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:09.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:09.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:24:09.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:24:09.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:24:09.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:24:09.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:24:09.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:24:09.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:24:09.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:24:10.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:24:10.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:24:10.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:10.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:10.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:24:10.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-22T20:24:10.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:24:10.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-22T20:24:10.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:24:10.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:24:10.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:24:10.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:24:10.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:24:10.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:24:10.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:24:10.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:24:10.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:24:10.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:24:10.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:24:10.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:24:10.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:10.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:10.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:24:10.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:24:10.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:24:10.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:24:10.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:24:10.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:24:10.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:24:10.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:24:10.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:24:10.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:24:10.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:24:10.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:24:10.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:24:10.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:24:11.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:24:11.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:24:11.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:24:11.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:24:11.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:24:11.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:24:11.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:24:11.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:24:11.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:24:11.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:24:11.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:11.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:11.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:24:11.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:24:11.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:24:11.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:24:11.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:24:11.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:24:11.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:24:11.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:24:11.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:24:11.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:24:11.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:24:11.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-22T20:24:11.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:24:11.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-22T20:24:12.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:24:12.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:24:12.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:24:12.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:24:12.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:24:12.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:24:12.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:24:12.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:24:12.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:24:12.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:24:12.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:12.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:12.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:24:12.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:24:12.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:24:12.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:24:12.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:24:12.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:24:12.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:24:12.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:24:12.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:24:12.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:24:12.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:24:12.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-22T20:24:12.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:24:12.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-22T20:24:13.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:24:13.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:24:13.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:24:13.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:24:13.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:24:13.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:24:13.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:24:13.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:24:13.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:24:13.520 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:24:13.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:13.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:13.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:24:13.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:24:13.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:24:13.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:630: TEST_auto_repair_bluestore_failed: get_pg testpool obj1 2026-02-22T20:24:13.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-22T20:24:13.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2026-02-22T20:24:13.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2026-02-22T20:24:13.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:24:13.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:630: TEST_auto_repair_bluestore_failed: local pgid=1.0 2026-02-22T20:24:13.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:631: TEST_auto_repair_bluestore_failed: get_primary testpool obj1 2026-02-22T20:24:13.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:24:13.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:24:13.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-22T20:24:13.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:24:14.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:631: TEST_auto_repair_bluestore_failed: local primary=1 2026-02-22T20:24:14.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed: get_last_scrub_stamp 1.0 2026-02-22T20:24:14.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:24:14.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:24:14.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:24:14.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:24:14.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:632: TEST_auto_repair_bluestore_failed: local last_scrub_stamp=2026-02-22T20:23:43.923245+0000 2026-02-22T20:24:14.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:633: TEST_auto_repair_bluestore_failed: ceph tell 1.0 schedule-deep-scrub 2026-02-22T20:24:14.266 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:24:14.266 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:24:14.266 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:24:14.266 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T15:36:04.261096+0000" 2026-02-22T20:24:14.266 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:24:14.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:636: TEST_auto_repair_bluestore_failed: wait_for_scrub 1.0 2026-02-22T20:23:43.923245+0000 2026-02-22T20:24:14.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:24:14.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:23:43.923245+0000 2026-02-22T20:24:14.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:24:14.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:24:14.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:24:14.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:24:14.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:24:14.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:24:14.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:24:14.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:24:14.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:23:43.923245+0000 '>' 2026-02-22T20:23:43.923245+0000 2026-02-22T20:24:14.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:24:15.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:24:15.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:24:15.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:24:15.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:24:15.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:24:15.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:24:15.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:24:15.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:23:43.923245+0000 '>' 2026-02-22T20:23:43.923245+0000 2026-02-22T20:24:15.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:24:16.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:24:16.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:24:16.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:24:16.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:24:16.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:24:16.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:24:16.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:24:16.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:23:43.923245+0000 '>' 2026-02-22T20:23:43.923245+0000 2026-02-22T20:24:16.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:24:17.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:24:17.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:24:17.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:24:17.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:24:17.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:24:17.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:24:17.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:24:17.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:16.084862+0000 '>' 2026-02-22T20:23:43.923245+0000 2026-02-22T20:24:17.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:24:17.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:637: TEST_auto_repair_bluestore_failed: wait_for_clean 2026-02-22T20:24:17.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:24:17.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:24:17.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:24:17.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:24:17.912 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:24:17.912 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:24:17.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:24:17.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:24:17.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:24:18.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:24:18.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:24:18.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:24:18.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:24:18.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:24:18.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:24:18.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:24:18.163 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:24:18.163 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:24:18.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:24:18.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:18.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:24:18.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084295 2026-02-22T20:24:18.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084295 2026-02-22T20:24:18.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084295' 2026-02-22T20:24:18.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:18.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:24:18.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888069 2026-02-22T20:24:18.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888069 2026-02-22T20:24:18.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084295 1-146028888069' 2026-02-22T20:24:18.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:18.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:24:18.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574862 2026-02-22T20:24:18.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574862 2026-02-22T20:24:18.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084295 1-146028888069 2-55834574862' 2026-02-22T20:24:18.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:18.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-120259084295 2026-02-22T20:24:18.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:18.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:24:18.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-120259084295 2026-02-22T20:24:18.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:18.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084295 2026-02-22T20:24:18.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 120259084295' 2026-02-22T20:24:18.386 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 120259084295 2026-02-22T20:24:18.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:18.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084294 -lt 120259084295 2026-02-22T20:24:18.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:24:19.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:24:19.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:19.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084295 -lt 120259084295 2026-02-22T20:24:19.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:19.704 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-146028888069 2026-02-22T20:24:19.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:19.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:24:19.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-146028888069 2026-02-22T20:24:19.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:19.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888069 2026-02-22T20:24:19.708 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 146028888069 2026-02-22T20:24:19.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 146028888069' 2026-02-22T20:24:19.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:24:19.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888069 -lt 146028888069 2026-02-22T20:24:19.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:19.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574862 2026-02-22T20:24:19.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:19.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:24:19.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574862 2026-02-22T20:24:19.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:19.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574862 2026-02-22T20:24:19.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574862' 2026-02-22T20:24:19.885 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574862 2026-02-22T20:24:19.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:24:20.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574862 -lt 55834574862 2026-02-22T20:24:20.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:24:20.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:20.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:20.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:24:20.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:24:20.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:24:20.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:24:20.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:24:20.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:24:20.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:24:20.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:24:20.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:24:20.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:24:20.387 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:20.387 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:24:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:24:20.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:24:20.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:638: TEST_auto_repair_bluestore_failed: flush_pg_stats 2026-02-22T20:24:20.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:24:20.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:24:20.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:24:20.736 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:24:20.736 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:24:20.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:24:20.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:20.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:24:20.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084296 2026-02-22T20:24:20.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084296 2026-02-22T20:24:20.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084296' 2026-02-22T20:24:20.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:20.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:24:20.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888070 2026-02-22T20:24:20.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888070 2026-02-22T20:24:20.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084296 1-146028888070' 2026-02-22T20:24:20.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:20.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:24:20.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574863 2026-02-22T20:24:20.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574863 2026-02-22T20:24:20.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084296 1-146028888070 2-55834574863' 2026-02-22T20:24:20.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:20.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-120259084296 2026-02-22T20:24:20.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:20.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:24:20.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-120259084296 2026-02-22T20:24:20.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:20.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084296 2026-02-22T20:24:20.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 120259084296' 2026-02-22T20:24:20.959 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 120259084296 2026-02-22T20:24:20.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:21.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084295 -lt 120259084296 2026-02-22T20:24:21.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:24:22.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:24:22.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:22.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084297 -lt 120259084296 2026-02-22T20:24:22.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:22.286 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-146028888070 2026-02-22T20:24:22.287 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:22.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:24:22.289 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-146028888070 2026-02-22T20:24:22.289 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:22.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888070 2026-02-22T20:24:22.290 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 146028888070 2026-02-22T20:24:22.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 146028888070' 2026-02-22T20:24:22.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:24:22.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888070 -lt 146028888070 2026-02-22T20:24:22.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:22.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574863 2026-02-22T20:24:22.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:22.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:24:22.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574863 2026-02-22T20:24:22.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:22.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574863 2026-02-22T20:24:22.462 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574863 2026-02-22T20:24:22.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574863' 2026-02-22T20:24:22.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:24:22.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574863 -lt 55834574863 2026-02-22T20:24:22.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:639: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-22T20:24:22.622 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.095+0000 7f487aa94640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-22T20:24:22.622 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.095+0000 7f487aa94640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-22T20:24:22.622 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.099+0000 7f487aa94640 15 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2026-02-22T20:24:22.622 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.099+0000 7f487aa94640 20 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: setting for deep-scrub-after-repair (5 errors. 2 errors fixed) 2026-02-22T20:24:22.622 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.099+0000 7f487aa94640 19 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-22T20:24:22.622 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:16.083+0000 7f4876a8c640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish before flags: CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-22T20:24:22.622 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:16.083+0000 7f4876a8c640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2026-02-22T20:24:22.622 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:16.083+0000 7f4876a8c640 19 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-22T20:24:22.622 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:16.083+0000 7f4876a8c640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair(after-repair)] scrubber: scrub_finish 3 error(s) still present after re-scrub 2026-02-22T20:24:22.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:640: TEST_auto_repair_bluestore_failed: grep -q 'scrub_finish.*still present after re-scrub' td/osd-scrub-repair/osd.1.log 2026-02-22T20:24:22.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:641: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2026-02-22T20:24:22.773 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:24:22.774 INFO:tasks.workunit.client.0.trial150.stdout:1.0 10 0 0 0 0 63 0 0 10 0 10 active+clean+inconsistent+failed_repair 2026-02-22T20:24:16.084912+0000 18'10 36:116 [1,0] 1 [1,0] 1 18'10 2026-02-22T20:24:16.084862+0000 18'10 2026-02-22T20:24:16.084862+0000 0 1 periodic scrub scheduled @ 2026-02-23T20:24:16.084862+0000 10 0 2026-02-22T20:24:22.774 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:24:22.774 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:24:22.774 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:24:22.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2026-02-22T20:24:22.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:642: TEST_auto_repair_bluestore_failed: grep -q '^1.0.*+failed_repair' 2026-02-22T20:24:22.945 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:24:22.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed: get_not_primary testpool obj1 2026-02-22T20:24:22.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:24:22.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-22T20:24:22.955 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2026-02-22T20:24:22.955 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:24:22.955 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:24:22.955 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-22T20:24:22.956 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:24:23.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:24:23.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2026-02-22T20:24:23.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:24:23.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:646: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 0 obj1 list-attrs 2026-02-22T20:24:23.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:24:23.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:24:23.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:24:23.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:24:23.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 list-attrs 2026-02-22T20:24:23.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:24:23.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:24:23.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:24:23.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:24:23.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:24:23.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:24:23.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:24:23.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:24:23.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:24:23.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:24:23.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:24:23.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 list-attrs 2026-02-22T20:24:23.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:24:23.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:24:23.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:24:23.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:24:23.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:24:23.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 list-attrs 2026-02-22T20:24:23.663 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:24:23.664 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:24:23.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:24:23.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:24:23.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:24:23.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:24:23.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:24:23.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:24:23.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:24:23.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:24:23.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:24:23.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:24:23.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:24:23.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:24:23.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:24:23.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:24:23.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:24:23.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:24:23.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:24:23.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:24:23.752 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:24:23.752 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:23.752 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:23.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:23.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:23.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:24:23.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:24:23.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:24:23.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:24:23.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:24:23.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:24:23.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:24:23.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:24:23.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:24:23.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:24:23.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:24:23.755 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:24:23.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:24:23.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:24:23.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:24:23.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:24:23.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:24:23.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:24:23.775 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:23.767+0000 7f053eb0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:23.777 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:23.771+0000 7f053eb0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:23.778 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:23.771+0000 7f053eb0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:23.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:24:23.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:24:23.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:24:23.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:24:23.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:24:23.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:23.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:24:23.916 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:24:23.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:23.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:24:24.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:24:24.158 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:24.151+0000 7f053eb0aa40 -1 Falling back to public interface 2026-02-22T20:24:24.756 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:24.747+0000 7f053eb0aa40 -1 osd.0 37 log_to_monitors true 2026-02-22T20:24:25.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:24:25.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:25.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:24:25.077 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:24:25.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:25.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:24:25.236 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 40 up_thru 31 down_at 37 last_clean_interval [28,37) [v2:127.0.0.1:6802/3858118807,v1:127.0.0.1:6803/3858118807] [v2:127.0.0.1:6804/3858118807,v1:127.0.0.1:6805/3858118807] exists,up 4e867cbb-1c13-4b64-9a4d-9c050ca0c6b6 2026-02-22T20:24:25.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:24:25.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:24:25.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:24:25.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:24:25.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:24:25.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:24:25.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:24:25.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:24:25.237 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:24:25.237 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:24:25.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:24:25.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:24:25.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:24:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:24:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:24:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:24:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:24:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:24:25.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:24:25.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:24:25.505 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:24:25.505 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:24:25.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:24:25.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:25.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:24:25.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691842 2026-02-22T20:24:25.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691842 2026-02-22T20:24:25.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691842' 2026-02-22T20:24:25.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:25.578 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:24:25.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888072 2026-02-22T20:24:25.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888072 2026-02-22T20:24:25.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691842 1-146028888072' 2026-02-22T20:24:25.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:25.651 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:24:25.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574865 2026-02-22T20:24:25.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574865 2026-02-22T20:24:25.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691842 1-146028888072 2-55834574865' 2026-02-22T20:24:25.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:25.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-171798691842 2026-02-22T20:24:25.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:25.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:24:25.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-171798691842 2026-02-22T20:24:25.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:25.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691842 2026-02-22T20:24:25.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 171798691842' 2026-02-22T20:24:25.727 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 171798691842 2026-02-22T20:24:25.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:25.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 171798691842 2026-02-22T20:24:25.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:24:26.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:24:26.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:27.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 171798691842 2026-02-22T20:24:27.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:24:28.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:24:28.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:28.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691842 -lt 171798691842 2026-02-22T20:24:28.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:28.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-146028888072 2026-02-22T20:24:28.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:28.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:24:28.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-146028888072 2026-02-22T20:24:28.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:28.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888072 2026-02-22T20:24:28.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 146028888072' 2026-02-22T20:24:28.219 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 146028888072 2026-02-22T20:24:28.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:24:28.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888073 -lt 146028888072 2026-02-22T20:24:28.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:28.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574865 2026-02-22T20:24:28.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:28.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:24:28.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574865 2026-02-22T20:24:28.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:28.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574865 2026-02-22T20:24:28.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574865' 2026-02-22T20:24:28.385 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 55834574865 2026-02-22T20:24:28.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:24:28.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574865 -lt 55834574865 2026-02-22T20:24:28.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:24:28.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:28.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:28.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:24:28.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:24:28.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:24:28.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:24:28.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:24:28.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:24:28.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:24:28.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:24:28.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:24:28.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:24:28.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:28.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:29.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:24:29.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:24:29.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:24:29.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:647: TEST_auto_repair_bluestore_failed: rados --pool testpool get obj1 td/osd-scrub-repair/COPY 2026-02-22T20:24:29.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:648: TEST_auto_repair_bluestore_failed: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:24:29.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:649: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-22T20:24:29.096 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.095+0000 7f487aa94640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-22T20:24:29.096 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.095+0000 7f487aa94640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-22T20:24:29.096 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.099+0000 7f487aa94640 15 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2026-02-22T20:24:29.096 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.099+0000 7f487aa94640 20 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: setting for deep-scrub-after-repair (5 errors. 2 errors fixed) 2026-02-22T20:24:29.097 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.099+0000 7f487aa94640 19 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-22T20:24:29.097 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:16.083+0000 7f4876a8c640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish before flags: CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-22T20:24:29.097 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:16.083+0000 7f4876a8c640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2026-02-22T20:24:29.097 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:16.083+0000 7f4876a8c640 19 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-22T20:24:29.097 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:16.083+0000 7f4876a8c640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair(after-repair)] scrubber: scrub_finish 3 error(s) still present after re-scrub 2026-02-22T20:24:29.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:652: TEST_auto_repair_bluestore_failed: get_primary testpool SOMETHING 2026-02-22T20:24:29.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:24:29.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:24:29.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:24:29.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:24:29.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:652: TEST_auto_repair_bluestore_failed: objectstore_tool td/osd-scrub-repair 1 obj2 remove 2026-02-22T20:24:29.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:24:29.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:24:29.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:24:29.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:24:29.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 remove 2026-02-22T20:24:29.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:24:29.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:24:29.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:24:29.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:24:29.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:24:29.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:24:29.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:24:29.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:24:29.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:24:29.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:24:29.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:24:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 remove 2026-02-22T20:24:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:24:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:24:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:24:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:24:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:24:29.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 remove 2026-02-22T20:24:29.757 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:104778fc:::obj2:head#, (61) No data available 2026-02-22T20:24:29.757 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:104778fc:::obj2:head# 2026-02-22T20:24:29.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:24:29.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:24:29.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:24:29.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:24:29.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:24:29.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:24:29.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:24:29.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:24:29.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:24:29.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:24:29.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:24:29.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:24:29.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:24:29.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:24:29.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:24:29.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:24:29.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:24:29.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:24:29.850 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:24:29.850 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:29.850 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:29.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:29.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:29.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:24:29.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:24:29.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:24:29.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:24:29.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:24:29.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:24:29.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:24:29.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:24:29.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:24:29.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:24:29.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:24:29.854 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:24:29.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:24:29.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:24:29.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:24:29.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:24:29.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:24:29.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:24:29.869 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:29.863+0000 7f9688f70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:29.870 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:29.863+0000 7f9688f70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:29.871 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:29.863+0000 7f9688f70a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:30.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:24:30.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:24:30.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:24:30.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:24:30.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:24:30.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:30.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:24:30.012 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:24:30.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:30.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:24:30.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:24:30.346 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:30.339+0000 7f9688f70a40 -1 Falling back to public interface 2026-02-22T20:24:30.971 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:30.963+0000 7f9688f70a40 -1 osd.1 41 log_to_monitors true 2026-02-22T20:24:31.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:24:31.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:31.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:24:31.166 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:24:31.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:31.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:24:31.321 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 42 last_clean_interval [34,41) [v2:127.0.0.1:6810/2421865893,v1:127.0.0.1:6811/2421865893] [v2:127.0.0.1:6812/2421865893,v1:127.0.0.1:6813/2421865893] exists,up 85c41e2e-5d63-4607-bded-8b0c5ae85a1c 2026-02-22T20:24:31.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:24:31.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:24:31.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:24:31.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:24:31.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:24:31.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:24:31.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:24:31.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:24:31.322 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:24:31.322 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:24:31.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:24:31.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:24:31.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:24:31.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:24:31.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:24:31.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:24:31.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:24:31.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:24:31.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:24:31.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:24:31.597 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:24:31.597 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:24:31.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:24:31.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:31.597 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:24:31.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691844 2026-02-22T20:24:31.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691844 2026-02-22T20:24:31.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691844' 2026-02-22T20:24:31.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:31.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:24:31.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528322 2026-02-22T20:24:31.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528322 2026-02-22T20:24:31.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691844 1-193273528322' 2026-02-22T20:24:31.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:31.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:24:31.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574867 2026-02-22T20:24:31.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574867 2026-02-22T20:24:31.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691844 1-193273528322 2-55834574867' 2026-02-22T20:24:31.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:31.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-171798691844 2026-02-22T20:24:31.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:31.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:24:31.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-171798691844 2026-02-22T20:24:31.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:31.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691844 2026-02-22T20:24:31.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 171798691844' 2026-02-22T20:24:31.820 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 171798691844 2026-02-22T20:24:31.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:31.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691843 -lt 171798691844 2026-02-22T20:24:31.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:24:32.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:24:32.987 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:33.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691843 -lt 171798691844 2026-02-22T20:24:33.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:24:34.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:24:34.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:34.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691844 -lt 171798691844 2026-02-22T20:24:34.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:34.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-193273528322 2026-02-22T20:24:34.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:34.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:24:34.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-193273528322 2026-02-22T20:24:34.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:34.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528322 2026-02-22T20:24:34.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 193273528322' 2026-02-22T20:24:34.315 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 193273528322 2026-02-22T20:24:34.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:24:34.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528322 -lt 193273528322 2026-02-22T20:24:34.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:34.475 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574867 2026-02-22T20:24:34.475 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:34.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:24:34.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574867 2026-02-22T20:24:34.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:34.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574867 2026-02-22T20:24:34.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574867' 2026-02-22T20:24:34.478 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 55834574867 2026-02-22T20:24:34.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:24:34.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574867 -lt 55834574867 2026-02-22T20:24:34.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:24:34.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:34.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:34.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:24:34.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:24:34.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:24:34.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:24:34.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:24:34.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:24:34.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:24:34.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:24:34.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:24:34.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:24:34.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:24:34.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:24:35.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:24:35.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:24:35.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:24:35.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:653: TEST_auto_repair_bluestore_failed: repair 1.0 2026-02-22T20:24:35.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-22T20:24:35.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-22T20:24:35.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:24:35.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:24:35.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:24:35.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:24:35.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:24:16.084862+0000 2026-02-22T20:24:35.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-22T20:24:35.492 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0 on osd.1 to repair 2026-02-22T20:24:35.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-22T20:24:16.084862+0000 2026-02-22T20:24:35.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:24:35.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:24:16.084862+0000 2026-02-22T20:24:35.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:24:35.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:24:35.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:24:35.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:24:35.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:24:35.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:24:35.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:24:35.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:24:35.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:16.084862+0000 '>' 2026-02-22T20:24:16.084862+0000 2026-02-22T20:24:35.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:24:36.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:24:36.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:24:36.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:24:36.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:24:36.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:24:36.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:24:36.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:24:36.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:16.084862+0000 '>' 2026-02-22T20:24:16.084862+0000 2026-02-22T20:24:36.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:24:37.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:24:37.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:24:37.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:24:37.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:24:37.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:24:37.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:24:37.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:24:37.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:16.084862+0000 '>' 2026-02-22T20:24:16.084862+0000 2026-02-22T20:24:37.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:24:38.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:24:38.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:24:38.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:24:38.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:24:38.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:24:38.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:24:38.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:24:39.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:16.084862+0000 '>' 2026-02-22T20:24:16.084862+0000 2026-02-22T20:24:39.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:24:40.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:24:40.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:24:40.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:24:40.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:24:40.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:24:40.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:24:40.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:24:40.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:16.084862+0000 '>' 2026-02-22T20:24:16.084862+0000 2026-02-22T20:24:40.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:24:41.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:24:41.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:24:41.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:24:41.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:24:41.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:24:41.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:24:41.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:24:41.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:36.087554+0000 '>' 2026-02-22T20:24:16.084862+0000 2026-02-22T20:24:41.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:24:41.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:654: TEST_auto_repair_bluestore_failed: sleep 2 2026-02-22T20:24:43.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:656: TEST_auto_repair_bluestore_failed: flush_pg_stats 2026-02-22T20:24:43.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:24:43.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:24:43.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:24:43.621 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:24:43.621 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:24:43.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:24:43.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:43.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:24:43.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691847 2026-02-22T20:24:43.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691847 2026-02-22T20:24:43.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691847' 2026-02-22T20:24:43.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:43.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:24:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528325 2026-02-22T20:24:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528325 2026-02-22T20:24:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691847 1-193273528325' 2026-02-22T20:24:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:24:43.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:24:43.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574871 2026-02-22T20:24:43.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574871 2026-02-22T20:24:43.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-171798691847 1-193273528325 2-55834574871' 2026-02-22T20:24:43.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:43.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-171798691847 2026-02-22T20:24:43.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:43.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:24:43.845 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-171798691847 2026-02-22T20:24:43.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:43.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691847 2026-02-22T20:24:43.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 171798691847' 2026-02-22T20:24:43.847 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 171798691847 2026-02-22T20:24:43.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:44.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691846 -lt 171798691847 2026-02-22T20:24:44.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:24:45.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:24:45.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:45.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691846 -lt 171798691847 2026-02-22T20:24:45.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:24:46.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:24:46.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:24:46.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691848 -lt 171798691847 2026-02-22T20:24:46.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:46.343 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-193273528325 2026-02-22T20:24:46.343 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:46.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:24:46.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-193273528325 2026-02-22T20:24:46.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:46.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528325 2026-02-22T20:24:46.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 193273528325' 2026-02-22T20:24:46.347 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 193273528325 2026-02-22T20:24:46.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:24:46.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528325 -lt 193273528325 2026-02-22T20:24:46.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:24:46.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574871 2026-02-22T20:24:46.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:24:46.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:24:46.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574871 2026-02-22T20:24:46.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:24:46.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574871 2026-02-22T20:24:46.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574871' 2026-02-22T20:24:46.516 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574871 2026-02-22T20:24:46.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:24:46.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574871 -lt 55834574871 2026-02-22T20:24:46.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:657: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2026-02-22T20:24:46.837 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:24:46.837 INFO:tasks.workunit.client.0.trial150.stdout:1.0 9 0 0 0 0 63 0 0 10 0 10 active+clean 2026-02-22T20:24:36.087620+0000 18'10 46:155 [1,0] 1 [1,0] 1 18'10 2026-02-22T20:24:36.087554+0000 18'10 2026-02-22T20:24:36.087554+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:24:36.087554+0000 9 0 2026-02-22T20:24:46.837 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:24:46.837 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:24:46.837 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:24:46.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:658: TEST_auto_repair_bluestore_failed: ceph pg dump pgs 2026-02-22T20:24:46.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:658: TEST_auto_repair_bluestore_failed: grep -q -e '^1.0.* active+clean ' -e '^1.0.* active+clean+wait ' 2026-02-22T20:24:46.995 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:24:47.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:659: TEST_auto_repair_bluestore_failed: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-22T20:24:47.008 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.095+0000 7f487aa94640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-22T20:24:47.008 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.095+0000 7f487aa94640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-22T20:24:47.008 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.099+0000 7f487aa94640 15 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 5 errors. 2 errors fixed 2026-02-22T20:24:47.008 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.099+0000 7f487aa94640 20 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: setting for deep-scrub-after-repair (5 errors. 2 errors fixed) 2026-02-22T20:24:47.008 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:15.099+0000 7f487aa94640 19 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/35/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-22T20:24:47.008 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:16.083+0000 7f4876a8c640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish before flags: CHECK_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-22T20:24:47.008 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:16.083+0000 7f4876a8c640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2026-02-22T20:24:47.008 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:16.083+0000 7f4876a8c640 19 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent(after-repair,check-repair)] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-22T20:24:47.008 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:16.083+0000 7f4876a8c640 10 osd.1 pg_epoch: 36 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=34/35 n=10 ec=14/14 lis/c=34/34 les/c/f=35/36/0 sis=34) [1,0] r=0 lpr=34 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+failed_repair(after-repair)] scrubber: scrub_finish 3 error(s) still present after re-scrub 2026-02-22T20:24:47.009 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:36.083+0000 7f966a6f5640 10 osd.1 pg_epoch: 46 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=45/46 n=10 ec=14/14 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: repair. deep_scrub_on_error: 0 2026-02-22T20:24:47.009 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:36.083+0000 7f966a6f5640 10 osd.1 pg_epoch: 46 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=45/46 n=10 ec=14/14 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-22T20:24:47.009 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:36.083+0000 7f966a6f5640 15 osd.1 pg_epoch: 46 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=45/46 n=9 ec=14/14 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: scrub_finish: 1 errors. 1 errors fixed 2026-02-22T20:24:47.009 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:36.083+0000 7f966a6f5640 20 osd.1 pg_epoch: 46 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=45/46 n=9 ec=14/14 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: scrub_finish All may be fixed 2026-02-22T20:24:47.009 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:24:36.083+0000 7f966a6f5640 19 osd.1 pg_epoch: 46 pg[1.0( v 18'10 (0'0,18'10] local-lis/les=45/46 n=9 ec=14/14 lis/c=45/45 les/c/f=46/46/0 sis=45) [1,0] r=0 lpr=45 crt=18'10 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+repair(must-repair,auto-repair)] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-22T20:24:47.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:24:47.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:24:47.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:24:47.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:24:47.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:24:47.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:24:47.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:24:47.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:24:47.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:24:47.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:24:47.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:24:47.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:24:47.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:24:47.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:24:47.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:24:47.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:24:47.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:24:47.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:24:47.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:24:47.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:24:47.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:24:47.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:24:47.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:24:47.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:24:47.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:47.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:47.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:24:47.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:24:47.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:24:47.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:24:47.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:24:47.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:24:47.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:24:47.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:24:47.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:24:47.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:24:47.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:24:47.145 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:24:47.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:24:47.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:24:47.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:24:47.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:24:47.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:24:47.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:24:47.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:24:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:24:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:24:47.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:24:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:24:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:24:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:24:47.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:24:47.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:24:47.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:24:47.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:24:47.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:24:47.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:47.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:47.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:24:47.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:24:47.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:24:47.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:24:47.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:24:47.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:47.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:47.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:24:47.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:24:47.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:24:47.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:24:47.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:24:47.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_failed_norecov td/osd-scrub-repair 2026-02-22T20:24:47.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:663: TEST_auto_repair_bluestore_failed_norecov: local dir=td/osd-scrub-repair 2026-02-22T20:24:47.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:664: TEST_auto_repair_bluestore_failed_norecov: local poolname=testpool 2026-02-22T20:24:47.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:667: TEST_auto_repair_bluestore_failed_norecov: run_mon td/osd-scrub-repair a 2026-02-22T20:24:47.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:24:47.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:24:47.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:24:47.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:24:47.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:24:47.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:24:47.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:24:47.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:24:47.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:24:47.186 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:24:47.186 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:47.186 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:47.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:47.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:24:47.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:24:47.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:24:47.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:24:47.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:24:47.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:24:47.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:24:47.218 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:24:47.218 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:24:47.218 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:24:47.218 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:24:47.218 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:47.219 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:47.219 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:24:47.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:24:47.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:24:47.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:24:47.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:24:47.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:24:47.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:24:47.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:24:47.274 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:24:47.274 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:24:47.274 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:24:47.275 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:24:47.275 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:47.275 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:47.275 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:24:47.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:24:47.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:24:47.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:668: TEST_auto_repair_bluestore_failed_norecov: run_mgr td/osd-scrub-repair x 2026-02-22T20:24:47.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:24:47.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:24:47.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:24:47.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:24:47.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:24:47.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:24:47.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:24:47.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:24:47.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:24:47.423 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:24:47.423 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:47.423 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:47.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:47.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:24:47.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:24:47.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:671: TEST_auto_repair_bluestore_failed_norecov: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:24:47.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_auto_repair_bluestore_failed_norecov: seq 0 2 2026-02-22T20:24:47.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2026-02-22T20:24:47.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:673: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:24:47.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:24:47.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:24:47.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:24:47.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:24:47.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:24:47.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:24:47.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:24:47.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:24:47.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:24:47.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:24:47.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:24:47.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:24:47.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:24:47.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:24:47.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:24:47.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:24:47.449 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:24:47.449 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:24:47.449 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:47.449 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:47.449 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:47.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:47.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:24:47.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:24:47.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:24:47.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:24:47.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:24:47.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:24:47.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:24:47.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:24:47.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:24:47.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:24:47.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:24:47.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:24:47.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:24:47.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=df8278f1-f682-4172-aa75-774dc79dd6ab 2026-02-22T20:24:47.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 df8278f1-f682-4172-aa75-774dc79dd6ab' 2026-02-22T20:24:47.453 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 df8278f1-f682-4172-aa75-774dc79dd6ab 2026-02-22T20:24:47.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:24:47.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAPZptp/lACGxAAMs47bvq89xJu/qDSSRdOwg== 2026-02-22T20:24:47.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAPZptp/lACGxAAMs47bvq89xJu/qDSSRdOwg=="}' 2026-02-22T20:24:47.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new df8278f1-f682-4172-aa75-774dc79dd6ab -i td/osd-scrub-repair/0/new.json 2026-02-22T20:24:47.547 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:24:47.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:24:47.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAPZptp/lACGxAAMs47bvq89xJu/qDSSRdOwg== --osd-uuid df8278f1-f682-4172-aa75-774dc79dd6ab 2026-02-22T20:24:47.573 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:47.567+0000 7f6909fa7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:47.575 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:47.567+0000 7f6909fa7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:47.576 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:47.567+0000 7f6909fa7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:47.576 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:47.567+0000 7f6909fa7a40 -1 bdev(0x561b41b48c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:24:47.576 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:47.567+0000 7f6909fa7a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:24:48.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:24:48.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:24:48.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:24:48.303 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:24:48.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:24:48.509 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:24:48.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:24:48.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:24:48.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:24:48.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:24:48.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:24:48.535 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:48.527+0000 7f9ee35c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:48.537 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:48.531+0000 7f9ee35c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:48.538 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:48.531+0000 7f9ee35c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:48.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:24:48.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:24:48.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:24:48.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:24:48.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:24:48.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:48.668 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:24:48.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:24:48.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:48.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:24:48.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:24:48.903 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:48.895+0000 7f9ee35c0a40 -1 Falling back to public interface 2026-02-22T20:24:49.536 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:49.527+0000 7f9ee35c0a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:24:49.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:24:49.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:49.825 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:24:49.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:24:49.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:49.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:24:49.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:24:50.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:24:50.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:50.981 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:24:50.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:24:50.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:50.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:24:51.142 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/873260867,v1:127.0.0.1:6803/873260867] [v2:127.0.0.1:6804/873260867,v1:127.0.0.1:6805/873260867] exists,up df8278f1-f682-4172-aa75-774dc79dd6ab 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:673: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:24:51.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:24:51.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:24:51.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:24:51.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:24:51.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:24:51.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:24:51.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:24:51.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:24:51.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:24:51.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:24:51.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:24:51.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f3025f1a-97a7-4a70-8a25-3f5fcb999c88 2026-02-22T20:24:51.147 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 f3025f1a-97a7-4a70-8a25-3f5fcb999c88 2026-02-22T20:24:51.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f3025f1a-97a7-4a70-8a25-3f5fcb999c88' 2026-02-22T20:24:51.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:24:51.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQATZptpQAEnCRAAsQykf10nfvmayxAlVddr9A== 2026-02-22T20:24:51.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQATZptpQAEnCRAAsQykf10nfvmayxAlVddr9A=="}' 2026-02-22T20:24:51.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f3025f1a-97a7-4a70-8a25-3f5fcb999c88 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:24:51.321 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:24:51.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:24:51.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQATZptpQAEnCRAAsQykf10nfvmayxAlVddr9A== --osd-uuid f3025f1a-97a7-4a70-8a25-3f5fcb999c88 2026-02-22T20:24:51.352 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:51.343+0000 7f81d40e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:51.354 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:51.347+0000 7f81d40e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:51.355 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:51.347+0000 7f81d40e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:51.355 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:51.347+0000 7f81d40e3a40 -1 bdev(0x55a6b1b84c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:24:51.355 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:51.347+0000 7f81d40e3a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:24:52.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:24:52.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:24:52.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:24:52.033 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:24:52.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:24:52.221 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:24:52.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:24:52.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:24:52.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:24:52.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:24:52.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:24:52.239 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:52.231+0000 7f05ad908a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:52.240 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:52.231+0000 7f05ad908a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:52.241 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:52.235+0000 7f05ad908a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:52.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:24:52.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:24:52.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:24:52.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:24:52.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:24:52.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:52.383 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:24:52.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:24:52.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:52.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:24:52.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:24:52.650 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:52.643+0000 7f05ad908a40 -1 Falling back to public interface 2026-02-22T20:24:53.201 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:53.195+0000 7f05ad908a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:24:53.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:24:53.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:53.544 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:24:53.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:24:53.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:53.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:24:53.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:24:54.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:24:54.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:54.707 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:24:54.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:24:54.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:54.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:24:54.865 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3931950250,v1:127.0.0.1:6811/3931950250] [v2:127.0.0.1:6812/3931950250,v1:127.0.0.1:6813/3931950250] exists,up f3025f1a-97a7-4a70-8a25-3f5fcb999c88 2026-02-22T20:24:54.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:24:54.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:24:54.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:24:54.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:672: TEST_auto_repair_bluestore_failed_norecov: for id in $(seq 0 2) 2026-02-22T20:24:54.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:673: TEST_auto_repair_bluestore_failed_norecov: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:24:54.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:24:54.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:24:54.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:24:54.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:24:54.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:24:54.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:24:54.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:24:54.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:24:54.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:24:54.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:24:54.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:24:54.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:24:54.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:24:54.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:24:54.871 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:24:54.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:24:54.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:24:54.872 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:24:54.872 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:24:54.872 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:24:54.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:54.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:24:54.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:24:54.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:24:54.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:24:54.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:24:54.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:24:54.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:24:54.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:24:54.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:24:54.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:24:54.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:24:54.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:24:54.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:24:54.873 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:24:54.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3e25b581-2a19-41ba-8142-dcd82868072b 2026-02-22T20:24:54.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 3e25b581-2a19-41ba-8142-dcd82868072b' 2026-02-22T20:24:54.874 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 3e25b581-2a19-41ba-8142-dcd82868072b 2026-02-22T20:24:54.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:24:54.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAWZptpx5M8NBAA0oxaMwssk7QMB43RxAZm8w== 2026-02-22T20:24:54.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAWZptpx5M8NBAA0oxaMwssk7QMB43RxAZm8w=="}' 2026-02-22T20:24:54.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3e25b581-2a19-41ba-8142-dcd82868072b -i td/osd-scrub-repair/2/new.json 2026-02-22T20:24:55.044 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:24:55.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:24:55.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQAWZptpx5M8NBAA0oxaMwssk7QMB43RxAZm8w== --osd-uuid 3e25b581-2a19-41ba-8142-dcd82868072b 2026-02-22T20:24:55.083 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:55.075+0000 7f348e161a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:55.084 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:55.075+0000 7f348e161a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:55.085 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:55.079+0000 7f348e161a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:55.086 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:55.079+0000 7f348e161a40 -1 bdev(0x55d23e5a2c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:24:55.087 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:55.079+0000 7f348e161a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:24:55.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:24:55.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:24:55.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:24:55.733 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:24:55.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:24:55.931 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:24:55.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:24:55.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:24:55.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:24:55.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:24:55.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:24:55.957 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:55.951+0000 7fcd83e19a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:55.959 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:55.951+0000 7fcd83e19a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:55.960 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:55.951+0000 7fcd83e19a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:24:56.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:24:56.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:24:56.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:24:56.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:24:56.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:24:56.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:56.089 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:24:56.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:24:56.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:56.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:24:56.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:24:56.343 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:56.335+0000 7fcd83e19a40 -1 Falling back to public interface 2026-02-22T20:24:56.958 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:24:56.951+0000 7fcd83e19a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:24:57.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:24:57.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:57.249 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:24:57.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:24:57.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:57.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:24:57.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:24:58.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:24:58.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:24:58.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:24:58.411 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:24:58.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:24:58.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:24:58.569 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/909915486,v1:127.0.0.1:6819/909915486] [v2:127.0.0.1:6820/909915486,v1:127.0.0.1:6821/909915486] exists,up 3e25b581-2a19-41ba-8142-dcd82868072b 2026-02-22T20:24:58.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:24:58.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:24:58.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:24:58.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:676: TEST_auto_repair_bluestore_failed_norecov: create_pool testpool 1 1 2026-02-22T20:24:58.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-22T20:24:58.777 INFO:tasks.workunit.client.0.trial150.stderr:pool 'testpool' created 2026-02-22T20:24:58.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:24:59.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:677: TEST_auto_repair_bluestore_failed_norecov: ceph osd pool set testpool size 2 2026-02-22T20:24:59.991 INFO:tasks.workunit.client.0.trial150.stderr:set pool 1 size to 2 2026-02-22T20:25:00.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:678: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2026-02-22T20:25:00.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:25:00.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:25:00.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:25:00.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:25:00.004 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:25:00.004 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:25:00.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:25:00.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:25:00.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:25:00.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:25:00.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:25:00.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:25:00.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:25:00.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:25:00.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:25:00.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:25:00.269 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:25:00.269 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:25:00.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:25:00.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:00.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:25:00.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:25:00.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:25:00.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:25:00.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:00.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:25:00.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-22T20:25:00.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-22T20:25:00.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-22T20:25:00.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:00.412 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:25:00.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509442 2026-02-22T20:25:00.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509442 2026-02-22T20:25:00.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-64424509442' 2026-02-22T20:25:00.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:00.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:25:00.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:00.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:25:00.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:25:00.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:00.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:25:00.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:25:00.489 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:25:00.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:00.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:25:00.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:25:01.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:25:01.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:01.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:25:01.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:25:02.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:25:02.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:02.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-22T20:25:02.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:02.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-22T20:25:02.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:02.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:25:02.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-22T20:25:02.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:02.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-22T20:25:02.982 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672963 2026-02-22T20:25:02.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-22T20:25:02.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:25:03.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-22T20:25:03.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:03.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509442 2026-02-22T20:25:03.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:03.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:25:03.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509442 2026-02-22T20:25:03.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:03.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509442 2026-02-22T20:25:03.152 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509442 2026-02-22T20:25:03.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2026-02-22T20:25:03.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:25:03.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509442 -lt 64424509442 2026-02-22T20:25:03.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:25:03.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:03.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:03.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:25:03.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:25:03.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:25:03.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:25:03.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:25:03.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:25:03.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:25:03.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:25:03.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:25:03.651 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:25:03.651 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:03.651 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:03.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:25:03.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-22T20:25:03.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:25:03.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-22T20:25:03.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:25:03.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:25:03.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:25:03.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:25:03.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:25:03.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:25:03.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:25:03.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:25:03.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:25:03.951 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:25:04.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:25:04.107 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:25:04.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:04.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:04.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:25:04.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:25:04.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:25:04.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:25:04.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:25:04.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:25:04.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:25:04.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:25:04.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:25:04.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:25:04.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:25:04.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:25:04.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:25:04.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:25:04.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:25:04.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:25:04.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:25:04.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:25:04.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:25:04.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:25:04.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:25:04.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:25:04.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:25:04.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:25:04.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:04.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:05.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:25:05.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:25:05.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:25:05.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:25:05.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:25:05.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:25:05.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:25:05.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:25:05.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:25:05.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:25:05.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:25:05.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-22T20:25:05.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:25:05.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-22T20:25:05.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:25:05.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:25:05.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:25:05.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:25:05.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:25:05.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:25:05.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:25:05.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:25:05.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:25:05.808 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:25:05.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:05.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:06.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:25:06.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:25:06.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:25:06.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:25:06.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:25:06.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:25:06.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:25:06.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:25:06.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:25:06.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:25:06.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:25:06.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-22T20:25:06.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:25:06.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-22T20:25:06.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:25:06.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:25:06.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:25:06.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:25:06.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:25:06.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:25:07.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:25:07.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:25:07.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:25:07.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:25:07.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:07.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:07.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:25:07.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:25:07.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:25:07.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:681: TEST_auto_repair_bluestore_failed_norecov: local payload=ABCDEF 2026-02-22T20:25:07.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:682: TEST_auto_repair_bluestore_failed_norecov: echo ABCDEF 2026-02-22T20:25:07.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: seq 1 10 2026-02-22T20:25:07.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-22T20:25:07.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:25:07.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-22T20:25:07.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:25:07.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-22T20:25:07.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:25:07.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-22T20:25:07.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:25:07.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-22T20:25:07.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:25:07.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-22T20:25:07.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:25:07.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-22T20:25:07.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:25:07.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-22T20:25:07.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:25:07.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-22T20:25:07.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:25:07.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:683: TEST_auto_repair_bluestore_failed_norecov: for i in $(seq 1 10) 2026-02-22T20:25:07.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:685: TEST_auto_repair_bluestore_failed_norecov: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:25:07.566 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2026-02-22T20:25:07.566 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:25:07.566 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:25:07.567 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-22T20:25:07.567 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:25:07.567 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:25:07.567 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:25:07.567 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:25:07.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:25:07.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:25:07.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:25:07.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:691: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj1 remove 2026-02-22T20:25:07.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:25:07.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:25:07.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:25:07.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:25:07.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj1 remove 2026-02-22T20:25:07.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:25:07.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:25:07.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:25:07.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:25:07.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:25:07.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:25:07.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:25:07.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:25:07.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:25:07.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:25:08.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:25:08.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj1 remove 2026-02-22T20:25:08.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:25:08.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:25:08.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:25:08.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:25:08.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:25:08.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj1 remove 2026-02-22T20:25:08.388 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-22T20:25:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:25:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:25:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:25:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:08.477 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:08.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:08.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:08.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:25:08.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:25:08.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:25:08.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:25:08.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:25:08.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:25:08.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:25:08.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:25:08.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:25:08.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:25:08.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:25:08.479 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:25:08.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:25:08.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:25:08.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:25:08.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:25:08.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:25:08.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:25:08.498 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:08.491+0000 7f18fe337a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:08.499 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:08.491+0000 7f18fe337a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:08.501 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:08.495+0000 7f18fe337a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:08.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:25:08.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:25:08.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:25:08.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:25:08.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:25:08.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:08.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:25:08.639 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:25:08.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:08.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:25:08.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:25:08.898 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:08.891+0000 7f18fe337a40 -1 Falling back to public interface 2026-02-22T20:25:09.489 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:09.483+0000 7f18fe337a40 -1 osd.0 21 log_to_monitors true 2026-02-22T20:25:09.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:25:09.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:09.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:25:09.802 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:25:09.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:09.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:25:09.961 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,21) [v2:127.0.0.1:6802/890844349,v1:127.0.0.1:6803/890844349] [v2:127.0.0.1:6804/890844349,v1:127.0.0.1:6805/890844349] exists,up df8278f1-f682-4172-aa75-774dc79dd6ab 2026-02-22T20:25:09.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:25:09.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:25:09.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:25:09.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:25:09.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:25:09.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:25:09.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:25:09.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:25:09.963 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:25:09.963 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:25:09.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:25:09.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:25:09.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:25:10.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:25:10.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:25:10.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:25:10.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:25:10.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:25:10.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:25:10.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:25:10.230 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:25:10.231 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:25:10.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:25:10.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:10.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:25:10.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-22T20:25:10.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-22T20:25:10.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-22T20:25:10.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:10.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:25:10.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672966 2026-02-22T20:25:10.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672966 2026-02-22T20:25:10.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-42949672966' 2026-02-22T20:25:10.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:10.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:25:10.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509445 2026-02-22T20:25:10.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509445 2026-02-22T20:25:10.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-42949672966 2-64424509445' 2026-02-22T20:25:10.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:10.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-22T20:25:10.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:10.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:25:10.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-22T20:25:10.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:10.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-22T20:25:10.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-22T20:25:10.451 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 103079215106 2026-02-22T20:25:10.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:10.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-22T20:25:10.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:25:11.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:25:11.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:11.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-22T20:25:11.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:25:12.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:25:12.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:12.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215106 2026-02-22T20:25:12.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:12.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672966 2026-02-22T20:25:12.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:12.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:25:12.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672966 2026-02-22T20:25:12.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:12.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672966 2026-02-22T20:25:12.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2026-02-22T20:25:12.942 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672966 2026-02-22T20:25:12.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:25:13.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672966 -lt 42949672966 2026-02-22T20:25:13.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:13.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509445 2026-02-22T20:25:13.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:13.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:25:13.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509445 2026-02-22T20:25:13.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509445 2026-02-22T20:25:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2026-02-22T20:25:13.106 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509445 2026-02-22T20:25:13.106 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:25:13.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509445 -lt 64424509445 2026-02-22T20:25:13.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:25:13.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:13.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:13.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:25:13.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:25:13.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:25:13.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:25:13.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:25:13.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:25:13.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:25:13.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:25:13.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:25:13.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:25:13.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:13.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:13.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:25:13.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:25:13.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:25:13.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:692: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2026-02-22T20:25:13.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:25:13.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:25:13.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:25:13.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:25:13.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:692: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj1 rm-attr _ 2026-02-22T20:25:13.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:25:13.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:25:13.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:25:13.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:25:13.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj1 rm-attr _ 2026-02-22T20:25:13.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:25:13.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:25:13.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:25:13.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:25:13.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:25:13.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:25:13.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:25:13.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:25:13.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:25:13.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:25:14.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:25:14.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 rm-attr _ 2026-02-22T20:25:14.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:25:14.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:25:14.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:25:14.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:25:14.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:25:14.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 rm-attr _ 2026-02-22T20:25:14.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:25:14.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:25:14.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:25:14.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:25:14.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:25:14.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:25:14.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:25:14.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:25:14.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:25:14.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:25:14.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:25:14.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:25:14.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:25:14.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:25:14.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:25:14.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:25:14.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:25:14.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:25:14.569 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:25:14.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:25:14.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:25:14.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:25:14.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:25:14.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:25:14.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:25:14.594 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:14.584+0000 7f9b9d37aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:14.596 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:14.588+0000 7f9b9d37aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:14.597 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:14.588+0000 7f9b9d37aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:14.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:25:14.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:25:14.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:25:14.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:25:14.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:25:14.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:14.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:25:14.732 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:25:14.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:14.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:25:14.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:25:15.001 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:14.992+0000 7f9b9d37aa40 -1 Falling back to public interface 2026-02-22T20:25:15.658 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:15.648+0000 7f9b9d37aa40 -1 osd.1 25 log_to_monitors true 2026-02-22T20:25:15.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:25:15.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:15.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:25:15.893 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:25:15.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:15.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:25:16.052 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [10,25) [v2:127.0.0.1:6810/797537059,v1:127.0.0.1:6811/797537059] [v2:127.0.0.1:6812/797537059,v1:127.0.0.1:6813/797537059] exists,up f3025f1a-97a7-4a70-8a25-3f5fcb999c88 2026-02-22T20:25:16.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:25:16.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:25:16.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:25:16.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:25:16.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:25:16.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:25:16.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:25:16.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:25:16.054 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:25:16.054 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:25:16.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:25:16.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:25:16.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:25:16.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:25:16.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:25:16.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:25:16.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:25:16.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:25:16.160 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:25:16.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:25:16.318 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:25:16.318 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:25:16.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:25:16.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:16.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:25:16.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215108 2026-02-22T20:25:16.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215108 2026-02-22T20:25:16.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108' 2026-02-22T20:25:16.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:16.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:25:16.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-22T20:25:16.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-22T20:25:16.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108 1-124554051586' 2026-02-22T20:25:16.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:16.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:25:16.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509447 2026-02-22T20:25:16.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509447 2026-02-22T20:25:16.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108 1-124554051586 2-64424509447' 2026-02-22T20:25:16.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:16.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215108 2026-02-22T20:25:16.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:16.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:25:16.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215108 2026-02-22T20:25:16.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:16.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215108 2026-02-22T20:25:16.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2026-02-22T20:25:16.547 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 103079215108 2026-02-22T20:25:16.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:16.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215107 -lt 103079215108 2026-02-22T20:25:16.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:25:17.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:25:17.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:17.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215107 -lt 103079215108 2026-02-22T20:25:17.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:25:18.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:25:18.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:19.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215108 -lt 103079215108 2026-02-22T20:25:19.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:19.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-124554051586 2026-02-22T20:25:19.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:19.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:25:19.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-124554051586 2026-02-22T20:25:19.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:19.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-22T20:25:19.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2026-02-22T20:25:19.039 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 124554051586 2026-02-22T20:25:19.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:25:19.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051586 -lt 124554051586 2026-02-22T20:25:19.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:19.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509447 2026-02-22T20:25:19.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:19.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:25:19.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509447 2026-02-22T20:25:19.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:19.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509447 2026-02-22T20:25:19.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2026-02-22T20:25:19.209 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509447 2026-02-22T20:25:19.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:25:19.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509447 -lt 64424509447 2026-02-22T20:25:19.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:25:19.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:19.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:19.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:25:19.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:25:19.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:25:19.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:25:19.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:25:19.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:25:19.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:25:19.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:25:19.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:25:19.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:25:19.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:19.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:19.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:25:19.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:25:19.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:25:19.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_auto_repair_bluestore_failed_norecov: get_not_primary testpool SOMETHING 2026-02-22T20:25:19.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:25:19.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:25:19.913 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-22T20:25:19.913 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:25:19.913 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:25:19.914 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:25:19.914 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:25:20.078 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:25:20.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:25:20.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:25:20.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:694: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 0 obj2 remove 2026-02-22T20:25:20.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:25:20.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:25:20.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:25:20.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:25:20.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 obj2 remove 2026-02-22T20:25:20.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:25:20.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:25:20.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:25:20.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:25:20.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:25:20.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:25:20.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:25:20.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:25:20.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:25:20.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:25:20.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:25:20.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2026-02-22T20:25:20.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:25:20.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:25:20.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:25:20.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:25:20.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:25:20.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2026-02-22T20:25:20.715 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:104778fc:::obj2:head# 2026-02-22T20:25:20.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:25:20.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:25:20.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:25:20.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:25:20.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:25:20.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:25:20.817 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:25:20.817 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:20.817 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:20.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:20.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:20.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:25:20.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:25:20.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:25:20.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:25:20.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:25:20.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:25:20.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:25:20.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:25:20.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:25:20.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:25:20.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:25:20.818 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:25:20.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:25:20.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:25:20.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:25:20.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:25:20.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:25:20.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:25:20.838 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:20.828+0000 7fdad2157a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:20.839 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:20.832+0000 7fdad2157a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:20.840 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:20.832+0000 7fdad2157a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:20.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:25:20.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:25:20.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:25:20.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:25:20.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:25:20.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:20.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:25:20.986 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:25:20.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:20.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:25:21.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:25:21.220 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:21.212+0000 7fdad2157a40 -1 Falling back to public interface 2026-02-22T20:25:21.760 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:21.752+0000 7fdad2157a40 -1 osd.0 30 log_to_monitors true 2026-02-22T20:25:22.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:25:22.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:22.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:25:22.147 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:25:22.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:22.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:25:22.304 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 34 up_thru 26 down_at 31 last_clean_interval [24,30) [v2:127.0.0.1:6802/1542827308,v1:127.0.0.1:6803/1542827308] [v2:127.0.0.1:6804/1542827308,v1:127.0.0.1:6805/1542827308] exists,up df8278f1-f682-4172-aa75-774dc79dd6ab 2026-02-22T20:25:22.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:25:22.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:25:22.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:25:22.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:25:22.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:25:22.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:25:22.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:25:22.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:25:22.306 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:25:22.306 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:25:22.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:25:22.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:25:22.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:25:22.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:25:22.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:25:22.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:25:22.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:25:22.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:25:22.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:25:22.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:25:22.571 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:25:22.571 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:25:22.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:25:22.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:22.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:25:22.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888066 2026-02-22T20:25:22.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888066 2026-02-22T20:25:22.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066' 2026-02-22T20:25:22.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:22.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:25:22.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051588 2026-02-22T20:25:22.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051588 2026-02-22T20:25:22.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066 1-124554051588' 2026-02-22T20:25:22.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:22.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:25:22.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509449 2026-02-22T20:25:22.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509449 2026-02-22T20:25:22.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066 1-124554051588 2-64424509449' 2026-02-22T20:25:22.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:22.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-146028888066 2026-02-22T20:25:22.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:22.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:25:22.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-146028888066 2026-02-22T20:25:22.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:22.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888066 2026-02-22T20:25:22.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2026-02-22T20:25:22.795 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 146028888066 2026-02-22T20:25:22.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:22.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 146028888066 2026-02-22T20:25:22.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:25:23.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:25:23.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:24.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888066 -lt 146028888066 2026-02-22T20:25:24.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:24.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-124554051588 2026-02-22T20:25:24.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:24.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:25:24.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-124554051588 2026-02-22T20:25:24.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:24.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051588 2026-02-22T20:25:24.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 124554051588' 2026-02-22T20:25:24.129 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 124554051588 2026-02-22T20:25:24.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:25:24.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051588 -lt 124554051588 2026-02-22T20:25:24.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:24.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509449 2026-02-22T20:25:24.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:24.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:25:24.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509449 2026-02-22T20:25:24.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:24.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509449 2026-02-22T20:25:24.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2026-02-22T20:25:24.298 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509449 2026-02-22T20:25:24.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:25:24.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509450 -lt 64424509449 2026-02-22T20:25:24.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:25:24.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:24.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:24.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:25:24.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:25:24.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:25:24.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:25:24.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:25:24.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:25:24.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:25:24.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:25:24.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:25:24.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:25:24.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:24.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:25.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:25:25.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:25:25.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:25:25.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool SOMETHING 2026-02-22T20:25:25.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:25:25.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:25:25.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:25:25.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:25:25.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:695: TEST_auto_repair_bluestore_failed_norecov: objectstore_tool td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-22T20:25:25.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:25:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:25:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:25:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:25:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-22T20:25:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:25:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:25:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:25:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:25:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:25:25.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:25:25.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:25:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:25:25.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:25:25.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:25:25.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:25:25.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 obj2 rm-attr _ 2026-02-22T20:25:25.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:25:25.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:25:25.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:25:25.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:25:25.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:25:25.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj2 rm-attr _ 2026-02-22T20:25:25.682 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:ff7b1f36:::obj1:head#, (61) No data available 2026-02-22T20:25:25.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:25:25.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:25:25.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:25:25.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:25:25.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:25:25.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:25:25.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:25:25.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:25:25.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:25:25.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:25:25.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:25:25.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:25:25.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:25:25.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:25:25.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:25:25.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:25:25.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:25:25.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:25:25.799 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:25:25.799 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:25.799 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:25.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:25.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:25.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:25:25.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:25:25.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:25:25.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:25:25.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:25:25.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:25:25.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:25:25.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:25:25.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:25:25.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:25:25.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:25:25.801 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:25:25.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:25:25.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:25:25.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:25:25.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:25:25.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:25:25.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:25:25.817 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:25.809+0000 7f321974ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:25.819 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:25.813+0000 7f321974ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:25.820 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:25.813+0000 7f321974ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:25.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:25:25.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:25:25.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:25:25.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:25:25.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:25:25.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:25.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:25:25.958 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:25:25.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:25.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:25:26.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:25:26.160 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:26.153+0000 7f321974ea40 -1 Falling back to public interface 2026-02-22T20:25:26.757 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:26.749+0000 7f321974ea40 -1 osd.1 35 log_to_monitors true 2026-02-22T20:25:27.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:25:27.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:27.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:25:27.117 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:25:27.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:27.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:25:27.273 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 39 up_thru 39 down_at 36 last_clean_interval [29,35) [v2:127.0.0.1:6810/4284978392,v1:127.0.0.1:6811/4284978392] [v2:127.0.0.1:6812/4284978392,v1:127.0.0.1:6813/4284978392] exists,up f3025f1a-97a7-4a70-8a25-3f5fcb999c88 2026-02-22T20:25:27.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:25:27.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:25:27.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:25:27.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:25:27.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:25:27.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:25:27.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:25:27.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:25:27.275 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:25:27.275 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:25:27.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:25:27.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:25:27.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:25:27.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:25:27.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:25:27.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:25:27.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:25:27.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:25:27.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:25:27.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:25:27.547 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:25:27.547 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:25:27.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:25:27.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:27.548 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:25:27.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888068 2026-02-22T20:25:27.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888068 2026-02-22T20:25:27.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888068' 2026-02-22T20:25:27.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:27.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:25:27.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724546 2026-02-22T20:25:27.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724546 2026-02-22T20:25:27.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888068 1-167503724546' 2026-02-22T20:25:27.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:27.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:25:27.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509451 2026-02-22T20:25:27.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509451 2026-02-22T20:25:27.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888068 1-167503724546 2-64424509451' 2026-02-22T20:25:27.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:27.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-146028888068 2026-02-22T20:25:27.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:27.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:25:27.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-146028888068 2026-02-22T20:25:27.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:27.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888068 2026-02-22T20:25:27.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 146028888068' 2026-02-22T20:25:27.764 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 146028888068 2026-02-22T20:25:27.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:27.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888066 -lt 146028888068 2026-02-22T20:25:27.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:25:28.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:25:28.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:29.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888068 -lt 146028888068 2026-02-22T20:25:29.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:29.094 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-167503724546 2026-02-22T20:25:29.094 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:29.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:25:29.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-167503724546 2026-02-22T20:25:29.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:29.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724546 2026-02-22T20:25:29.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 167503724546' 2026-02-22T20:25:29.098 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 167503724546 2026-02-22T20:25:29.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:25:29.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724546 -lt 167503724546 2026-02-22T20:25:29.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:29.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509451 2026-02-22T20:25:29.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:29.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:25:29.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509451 2026-02-22T20:25:29.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:29.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509451 2026-02-22T20:25:29.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2026-02-22T20:25:29.272 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509451 2026-02-22T20:25:29.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:25:29.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509451 -lt 64424509451 2026-02-22T20:25:29.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:25:29.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:29.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:29.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:25:29.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:25:29.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:25:29.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:25:29.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:25:29.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:25:29.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:25:29.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:25:29.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:25:29.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:25:29.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:29.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:29.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:25:29.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:25:29.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:25:29.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:696: TEST_auto_repair_bluestore_failed_norecov: ceph tell 'osd.*' config set osd_scrub_auto_repair true 2026-02-22T20:25:30.054 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:25:30.055 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:25:30.055 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:25:30.062 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:25:30.062 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_auto_repair = '' (not observed, change may require restart) osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:25:30.062 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:25:30.069 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:25:30.069 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_auto_repair = '' (not observed, change may require restart) " 2026-02-22T20:25:30.069 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:25:30.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_auto_repair_bluestore_failed_norecov: get_pg testpool obj1 2026-02-22T20:25:30.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-22T20:25:30.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2026-02-22T20:25:30.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2026-02-22T20:25:30.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:25:30.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:698: TEST_auto_repair_bluestore_failed_norecov: local pgid=1.0 2026-02-22T20:25:30.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:699: TEST_auto_repair_bluestore_failed_norecov: get_primary testpool obj1 2026-02-22T20:25:30.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:25:30.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:25:30.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-22T20:25:30.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:25:30.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:699: TEST_auto_repair_bluestore_failed_norecov: local primary=1 2026-02-22T20:25:30.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_auto_repair_bluestore_failed_norecov: get_last_scrub_stamp 1.0 2026-02-22T20:25:30.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:25:30.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:25:30.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:25:30.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:25:30.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:700: TEST_auto_repair_bluestore_failed_norecov: local last_scrub_stamp=2026-02-22T20:24:58.771202+0000 2026-02-22T20:25:30.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:701: TEST_auto_repair_bluestore_failed_norecov: ceph tell 1.0 schedule-deep-scrub 2026-02-22T20:25:30.631 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:25:30.631 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:25:30.631 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:25:30.631 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T15:37:20.628079+0000" 2026-02-22T20:25:30.631 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:25:30.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:704: TEST_auto_repair_bluestore_failed_norecov: wait_for_scrub 1.0 2026-02-22T20:24:58.771202+0000 2026-02-22T20:25:30.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:25:30.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:24:58.771202+0000 2026-02-22T20:25:30.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:25:30.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:25:30.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:25:30.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:25:30.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:25:30.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:25:30.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:25:30.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:25:30.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:58.771202+0000 '>' 2026-02-22T20:24:58.771202+0000 2026-02-22T20:25:30.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:25:31.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:25:31.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:25:31.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:25:31.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:25:31.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:25:31.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:25:31.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:25:31.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:58.771202+0000 '>' 2026-02-22T20:24:58.771202+0000 2026-02-22T20:25:31.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:25:32.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:25:32.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:25:32.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:25:32.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:25:32.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:25:32.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:25:32.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:25:33.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:58.771202+0000 '>' 2026-02-22T20:24:58.771202+0000 2026-02-22T20:25:33.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:25:34.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:25:34.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:25:34.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:25:34.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:25:34.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:25:34.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:25:34.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:25:34.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:58.771202+0000 '>' 2026-02-22T20:24:58.771202+0000 2026-02-22T20:25:34.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:25:35.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:25:35.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:25:35.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:25:35.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:25:35.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:25:35.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:25:35.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:25:35.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:58.771202+0000 '>' 2026-02-22T20:24:58.771202+0000 2026-02-22T20:25:35.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:25:36.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:25:36.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:25:36.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:25:36.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:25:36.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:25:36.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:25:36.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:25:36.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:58.771202+0000 '>' 2026-02-22T20:24:58.771202+0000 2026-02-22T20:25:36.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:25:37.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:25:37.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:25:37.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:25:37.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:25:37.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:25:37.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:25:37.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:25:37.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:24:58.771202+0000 '>' 2026-02-22T20:24:58.771202+0000 2026-02-22T20:25:37.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:25:38.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:25:38.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:25:38.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:25:38.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:25:38.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:25:38.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:25:38.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:25:38.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:25:32.749992+0000 '>' 2026-02-22T20:24:58.771202+0000 2026-02-22T20:25:38.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:25:38.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:705: TEST_auto_repair_bluestore_failed_norecov: wait_for_clean 2026-02-22T20:25:38.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:25:38.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:25:38.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:25:38.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:25:38.904 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:25:38.904 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:25:38.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:25:38.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:25:38.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:25:39.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:25:39.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:25:39.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:25:39.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:25:39.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:25:39.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:25:39.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:25:39.172 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:25:39.172 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:25:39.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:25:39.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:39.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:25:39.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888071 2026-02-22T20:25:39.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888071 2026-02-22T20:25:39.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888071' 2026-02-22T20:25:39.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:39.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:25:39.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724549 2026-02-22T20:25:39.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724549 2026-02-22T20:25:39.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888071 1-167503724549' 2026-02-22T20:25:39.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:39.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:25:39.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509455 2026-02-22T20:25:39.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509455 2026-02-22T20:25:39.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888071 1-167503724549 2-64424509455' 2026-02-22T20:25:39.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:39.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-146028888071 2026-02-22T20:25:39.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:39.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:25:39.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-146028888071 2026-02-22T20:25:39.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:39.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888071 2026-02-22T20:25:39.395 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 146028888071 2026-02-22T20:25:39.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 146028888071' 2026-02-22T20:25:39.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:39.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888070 -lt 146028888071 2026-02-22T20:25:39.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:25:40.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:25:40.566 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:40.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888071 -lt 146028888071 2026-02-22T20:25:40.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:40.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-167503724549 2026-02-22T20:25:40.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:40.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:25:40.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-167503724549 2026-02-22T20:25:40.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:40.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724549 2026-02-22T20:25:40.726 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 167503724549 2026-02-22T20:25:40.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 167503724549' 2026-02-22T20:25:40.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:25:40.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724549 -lt 167503724549 2026-02-22T20:25:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:40.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509455 2026-02-22T20:25:40.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:40.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:25:40.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509455 2026-02-22T20:25:40.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:40.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509455 2026-02-22T20:25:40.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2026-02-22T20:25:40.900 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509455 2026-02-22T20:25:40.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:25:41.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509455 -lt 64424509455 2026-02-22T20:25:41.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:25:41.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:41.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:41.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:25:41.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:25:41.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:25:41.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:25:41.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:25:41.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:25:41.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:25:41.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:25:41.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:25:41.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:25:41.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:25:41.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:25:41.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:25:41.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:25:41.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:25:41.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:706: TEST_auto_repair_bluestore_failed_norecov: flush_pg_stats 2026-02-22T20:25:41.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:25:41.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:25:41.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:25:41.762 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:25:41.762 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:25:41.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:25:41.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:41.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:25:41.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888072 2026-02-22T20:25:41.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888072 2026-02-22T20:25:41.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888072' 2026-02-22T20:25:41.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:41.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:25:41.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724551 2026-02-22T20:25:41.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724551 2026-02-22T20:25:41.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888072 1-167503724551' 2026-02-22T20:25:41.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:41.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:25:41.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509456 2026-02-22T20:25:41.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509456 2026-02-22T20:25:41.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888072 1-167503724551 2-64424509456' 2026-02-22T20:25:41.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:41.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-146028888072 2026-02-22T20:25:41.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:41.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:25:41.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-146028888072 2026-02-22T20:25:41.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:41.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888072 2026-02-22T20:25:41.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 146028888072' 2026-02-22T20:25:41.981 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 146028888072 2026-02-22T20:25:41.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:42.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888073 -lt 146028888072 2026-02-22T20:25:42.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:42.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-167503724551 2026-02-22T20:25:42.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:42.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:25:42.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-167503724551 2026-02-22T20:25:42.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:42.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724551 2026-02-22T20:25:42.153 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 167503724551 2026-02-22T20:25:42.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 167503724551' 2026-02-22T20:25:42.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:25:42.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724551 -lt 167503724551 2026-02-22T20:25:42.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:42.327 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509456 2026-02-22T20:25:42.327 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:42.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:25:42.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509456 2026-02-22T20:25:42.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:42.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509456 2026-02-22T20:25:42.330 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509456 2026-02-22T20:25:42.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2026-02-22T20:25:42.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:25:42.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509455 -lt 64424509456 2026-02-22T20:25:42.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:25:43.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:25:43.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:25:43.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509455 -lt 64424509456 2026-02-22T20:25:43.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:25:44.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:25:44.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:25:44.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509457 -lt 64424509456 2026-02-22T20:25:44.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:707: TEST_auto_repair_bluestore_failed_norecov: grep -q 'scrub_finish.*present with no repair possible' td/osd-scrub-repair/osd.1.log 2026-02-22T20:25:44.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:708: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2026-02-22T20:25:44.978 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:25:44.978 INFO:tasks.workunit.client.0.trial150.stdout:1.0 10 0 0 0 0 56 0 0 10 0 10 active+clean+inconsistent+failed_repair 2026-02-22T20:25:32.750072+0000 20'10 40:109 [1,0] 1 [1,0] 1 20'10 2026-02-22T20:25:32.749992+0000 20'10 2026-02-22T20:25:32.749992+0000 0 0 periodic scrub scheduled @ 2026-02-23T20:25:32.749992+0000 10 0 2026-02-22T20:25:44.979 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:25:44.979 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:25:44.979 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:25:44.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:709: TEST_auto_repair_bluestore_failed_norecov: ceph pg dump pgs 2026-02-22T20:25:44.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:709: TEST_auto_repair_bluestore_failed_norecov: grep -q '^1.0.*+failed_repair' 2026-02-22T20:25:45.135 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:25:45.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:25:45.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:25:45.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:25:45.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:25:45.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:25:45.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:25:45.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:25:45.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:25:45.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:25:45.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:25:45.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:25:45.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:25:45.459 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:25:45.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:25:45.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:25:45.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:25:45.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:25:45.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:25:45.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:25:45.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:25:45.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:25:45.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:25:45.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:25:45.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:25:45.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:45.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:45.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:25:45.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:25:45.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:25:45.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:25:45.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:25:45.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:25:45.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:25:45.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:25:45.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:25:45.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:25:45.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:25:45.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:25:45.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:25:45.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:25:45.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:25:45.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:25:45.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:25:45.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:25:45.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:25:45.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:25:45.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:25:45.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:25:45.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:25:45.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:25:45.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:25:45.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:25:45.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:25:45.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:25:45.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:25:45.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:25:45.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:45.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:45.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:25:45.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:25:45.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:25:45.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:25:45.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:25:45.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:45.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:45.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:25:45.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_scrub td/osd-scrub-repair 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:542: TEST_auto_repair_bluestore_scrub: local dir=td/osd-scrub-repair 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:543: TEST_auto_repair_bluestore_scrub: local poolname=testpool 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:546: TEST_auto_repair_bluestore_scrub: run_mon td/osd-scrub-repair a 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:25:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:25:45.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:25:45.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:25:45.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:25:45.522 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:25:45.522 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:45.522 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:45.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:45.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:25:45.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:25:45.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:25:45.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:25:45.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:25:45.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:25:45.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:25:45.552 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:25:45.552 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:25:45.552 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:25:45.552 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:25:45.552 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:45.552 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:45.553 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:25:45.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:25:45.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:25:45.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:25:45.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:25:45.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:25:45.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:25:45.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:25:45.610 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:25:45.610 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:25:45.610 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:25:45.610 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:25:45.610 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:45.610 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:45.611 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:25:45.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:25:45.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:25:45.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:547: TEST_auto_repair_bluestore_scrub: run_mgr td/osd-scrub-repair x 2026-02-22T20:25:45.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:25:45.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:25:45.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:25:45.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:25:45.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:25:45.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:25:45.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:25:45.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:25:45.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:25:45.759 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:25:45.759 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:45.759 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:45.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:45.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:25:45.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:25:45.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:551: TEST_auto_repair_bluestore_scrub: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-22T20:25:45.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_scrub: seq 0 2 2026-02-22T20:25:45.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2026-02-22T20:25:45.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-22T20:25:45.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:25:45.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:25:45.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:25:45.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:25:45.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:25:45.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:25:45.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:25:45.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:25:45.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:25:45.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:25:45.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:25:45.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:25:45.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:25:45.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:25:45.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:25:45.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:25:45.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:25:45.786 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:25:45.786 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:45.786 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:45.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:45.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:45.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:25:45.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:25:45.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:25:45.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:25:45.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:25:45.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:25:45.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:25:45.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:25:45.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:25:45.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:25:45.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-22T20:25:45.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:25:45.787 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:25:45.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=318617e4-01ea-4819-ace4-a5a4256a74b1 2026-02-22T20:25:45.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 318617e4-01ea-4819-ace4-a5a4256a74b1' 2026-02-22T20:25:45.788 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 318617e4-01ea-4819-ace4-a5a4256a74b1 2026-02-22T20:25:45.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:25:45.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBJZptpY+NILxAAIQRC4BN68DriaR4ZPX1x6w== 2026-02-22T20:25:45.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBJZptpY+NILxAAIQRC4BN68DriaR4ZPX1x6w=="}' 2026-02-22T20:25:45.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 318617e4-01ea-4819-ace4-a5a4256a74b1 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:25:45.886 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:25:45.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:25:45.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQBJZptpY+NILxAAIQRC4BN68DriaR4ZPX1x6w== --osd-uuid 318617e4-01ea-4819-ace4-a5a4256a74b1 2026-02-22T20:25:45.913 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:45.905+0000 7f746a892a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:45.914 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:45.909+0000 7f746a892a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:45.915 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:45.909+0000 7f746a892a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:45.916 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:45.909+0000 7f746a892a40 -1 bdev(0x55a569ed0c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:25:45.916 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:45.909+0000 7f746a892a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:25:46.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:25:46.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:25:46.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:25:46.650 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:25:46.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:25:46.851 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:25:46.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:25:46.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-22T20:25:46.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:25:46.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:25:46.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:25:46.877 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:46.869+0000 7f175260aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:46.879 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:46.873+0000 7f175260aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:46.880 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:46.873+0000 7f175260aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:47.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:25:47.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:25:47.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:25:47.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:25:47.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:25:47.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:47.010 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:25:47.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:25:47.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:47.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:25:47.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:25:47.260 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:47.253+0000 7f175260aa40 -1 Falling back to public interface 2026-02-22T20:25:47.814 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:47.805+0000 7f175260aa40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:25:48.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:25:48.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:48.174 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:25:48.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:25:48.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:48.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:25:48.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:25:49.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:25:49.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:49.333 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:25:49.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:25:49.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:49.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:25:49.492 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3836341401,v1:127.0.0.1:6803/3836341401] [v2:127.0.0.1:6804/3836341401,v1:127.0.0.1:6805/3836341401] exists,up 318617e4-01ea-4819-ace4-a5a4256a74b1 2026-02-22T20:25:49.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:25:49.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:25:49.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:25:49.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2026-02-22T20:25:49.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-22T20:25:49.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:25:49.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:25:49.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:49.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-22T20:25:49.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:25:49.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:25:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e1987e33-a01d-4988-9b17-a8e0477bf904 2026-02-22T20:25:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e1987e33-a01d-4988-9b17-a8e0477bf904' 2026-02-22T20:25:49.497 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 e1987e33-a01d-4988-9b17-a8e0477bf904 2026-02-22T20:25:49.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:25:49.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBNZptpuN4zHhAAJeb1817DBnt2YdL4PKKNzA== 2026-02-22T20:25:49.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBNZptpuN4zHhAAJeb1817DBnt2YdL4PKKNzA=="}' 2026-02-22T20:25:49.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e1987e33-a01d-4988-9b17-a8e0477bf904 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:25:49.678 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:25:49.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:25:49.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQBNZptpuN4zHhAAJeb1817DBnt2YdL4PKKNzA== --osd-uuid e1987e33-a01d-4988-9b17-a8e0477bf904 2026-02-22T20:25:49.705 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:49.698+0000 7ff119e95a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:49.706 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:49.702+0000 7ff119e95a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:49.707 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:49.702+0000 7ff119e95a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:49.708 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:49.702+0000 7ff119e95a40 -1 bdev(0x55d971080c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:25:49.708 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:49.702+0000 7ff119e95a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:25:50.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:25:50.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:25:50.420 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:25:50.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:25:50.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:25:50.613 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:25:50.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:25:50.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-22T20:25:50.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:25:50.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:25:50.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:25:50.631 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:50.626+0000 7f8354fada40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:50.632 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:50.626+0000 7f8354fada40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:50.633 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:50.626+0000 7f8354fada40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:50.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:25:50.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:25:50.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:25:50.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:25:50.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:25:50.770 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:25:50.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:50.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:25:50.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:50.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:25:50.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:25:51.056 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:51.050+0000 7f8354fada40 -1 Falling back to public interface 2026-02-22T20:25:51.656 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:51.650+0000 7f8354fada40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:25:51.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:25:51.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:51.930 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:25:51.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:25:51.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:51.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:25:52.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:25:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:25:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:25:53.091 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:25:53.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:25:53.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:53.250 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1493397334,v1:127.0.0.1:6811/1493397334] [v2:127.0.0.1:6812/1493397334,v1:127.0.0.1:6813/1493397334] exists,up e1987e33-a01d-4988-9b17-a8e0477bf904 2026-02-22T20:25:53.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:25:53.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:25:53.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:25:53.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:552: TEST_auto_repair_bluestore_scrub: for id in $(seq 0 2) 2026-02-22T20:25:53.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:553: TEST_auto_repair_bluestore_scrub: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-22T20:25:53.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:25:53.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:25:53.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:25:53.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:25:53.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:25:53.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:25:53.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:25:53.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:25:53.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:25:53.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:25:53.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:25:53.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:25:53.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:25:53.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:25:53.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:25:53.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:25:53.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:25:53.255 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:25:53.255 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:25:53.255 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:25:53.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:53.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:25:53.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:25:53.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:25:53.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:25:53.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:25:53.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:25:53.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:25:53.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:25:53.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:25:53.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:25:53.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:25:53.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-22T20:25:53.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:25:53.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:25:53.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6681ca19-1a91-474f-9134-b638018207fb 2026-02-22T20:25:53.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6681ca19-1a91-474f-9134-b638018207fb' 2026-02-22T20:25:53.257 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 6681ca19-1a91-474f-9134-b638018207fb 2026-02-22T20:25:53.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:25:53.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBRZptp8H/JDxAAMxBNJCg+L7Vw1eq9MCuuoQ== 2026-02-22T20:25:53.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBRZptp8H/JDxAAMxBNJCg+L7Vw1eq9MCuuoQ=="}' 2026-02-22T20:25:53.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6681ca19-1a91-474f-9134-b638018207fb -i td/osd-scrub-repair/2/new.json 2026-02-22T20:25:53.421 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:25:53.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:25:53.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 --mkfs --key AQBRZptp8H/JDxAAMxBNJCg+L7Vw1eq9MCuuoQ== --osd-uuid 6681ca19-1a91-474f-9134-b638018207fb 2026-02-22T20:25:53.449 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:53.442+0000 7f469c1c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:53.451 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:53.446+0000 7f469c1c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:53.452 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:53.446+0000 7f469c1c8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:53.452 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:53.446+0000 7f469c1c8a40 -1 bdev(0x55d029f4ec00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:25:53.452 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:53.446+0000 7f469c1c8a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:25:54.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:25:54.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:25:54.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:25:54.090 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:25:54.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:25:54.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:25:54.287 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:25:54.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-22T20:25:54.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:25:54.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:25:54.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:25:54.304 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:54.298+0000 7f929fc9ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:54.305 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:54.298+0000 7f929fc9ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:54.306 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:54.302+0000 7f929fc9ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:25:54.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:25:54.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:25:54.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:25:54.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:25:54.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:25:54.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:54.445 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:25:54.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:25:54.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:54.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:25:54.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:25:54.743 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:54.738+0000 7f929fc9ea40 -1 Falling back to public interface 2026-02-22T20:25:55.384 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:25:55.378+0000 7f929fc9ea40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:25:55.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:25:55.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:55.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:25:55.605 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:25:55.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:55.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:25:55.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:25:56.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:25:56.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:25:56.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:25:56.760 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:25:56.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:25:56.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:25:56.921 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4078948043,v1:127.0.0.1:6819/4078948043] [v2:127.0.0.1:6820/4078948043,v1:127.0.0.1:6821/4078948043] exists,up 6681ca19-1a91-474f-9134-b638018207fb 2026-02-22T20:25:56.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:25:56.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:25:56.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:25:56.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:556: TEST_auto_repair_bluestore_scrub: create_pool testpool 1 1 2026-02-22T20:25:56.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-22T20:25:57.123 INFO:tasks.workunit.client.0.trial150.stderr:pool 'testpool' created 2026-02-22T20:25:57.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:25:58.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:557: TEST_auto_repair_bluestore_scrub: ceph osd pool set testpool size 2 2026-02-22T20:25:58.333 INFO:tasks.workunit.client.0.trial150.stderr:set pool 1 size to 2 2026-02-22T20:25:58.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:558: TEST_auto_repair_bluestore_scrub: wait_for_clean 2026-02-22T20:25:58.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:25:58.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:25:58.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:25:58.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:25:58.345 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:25:58.346 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:25:58.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:25:58.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:25:58.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:25:58.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:25:58.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:25:58.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:25:58.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:25:58.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:25:58.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:25:58.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:25:58.620 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:25:58.620 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:25:58.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:25:58.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:58.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:25:58.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:25:58.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:25:58.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:25:58.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:58.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:25:58.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-22T20:25:58.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-22T20:25:58.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-22T20:25:58.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:25:58.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:25:58.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-22T20:25:58.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-22T20:25:58.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-60129542146' 2026-02-22T20:25:58.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:25:58.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:25:58.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:25:58.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:25:58.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:25:58.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:25:58.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:25:58.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:25:58.842 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:25:58.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:25:59.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:25:59.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:26:00.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:26:00.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:00.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:26:00.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:26:01.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:26:01.164 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:01.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-22T20:26:01.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:01.327 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-22T20:26:01.327 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:01.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:26:01.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-22T20:26:01.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:01.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-22T20:26:01.330 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672963 2026-02-22T20:26:01.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-22T20:26:01.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:26:01.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-22T20:26:01.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:01.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-22T20:26:01.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:01.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:26:01.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-22T20:26:01.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:01.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-22T20:26:01.501 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542146 2026-02-22T20:26:01.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-22T20:26:01.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:26:01.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-22T20:26:01.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:26:01.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:01.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:01.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:26:01.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:26:01.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:26:01.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:26:01.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:26:01.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:26:01.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:26:01.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:26:02.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:26:02.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:26:02.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:02.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:02.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:26:02.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:26:02.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:26:02.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:561: TEST_auto_repair_bluestore_scrub: local payload=ABCDEF 2026-02-22T20:26:02.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:562: TEST_auto_repair_bluestore_scrub: echo ABCDEF 2026-02-22T20:26:02.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:563: TEST_auto_repair_bluestore_scrub: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:26:02.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:567: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2026-02-22T20:26:02.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:26:02.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:26:02.232 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-22T20:26:02.232 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:26:02.232 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:26:02.233 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:26:02.233 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:26:02.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:26:02.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:26:02.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:26:02.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:567: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:26:02.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:26:02.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:26:02.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:26:02.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:26:02.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:26:02.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:26:02.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:26:02.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:26:02.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:26:02.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:26:02.552 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:26:02.552 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:26:02.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:26:02.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:26:02.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:26:02.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:26:02.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:26:02.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:26:02.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:26:02.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:26:02.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:26:02.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:26:02.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-22T20:26:03.058 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-22T20:26:03.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-22T20:26:03.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:26:03.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:26:03.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:26:03.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:26:03.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:26:03.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:26:03.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:26:03.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:26:03.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:26:03.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:26:03.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:26:03.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:26:03.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:26:03.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:26:03.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:26:03.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:26:03.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:26:03.159 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:26:03.159 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:03.159 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:03.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:03.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:03.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:26:03.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:26:03.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:26:03.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:26:03.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:26:03.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:26:03.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:26:03.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:26:03.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-22T20:26:03.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:26:03.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:26:03.161 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:26:03.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-22T20:26:03.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:26:03.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:26:03.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:26:03.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:26:03.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:26:03.178 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:03.174+0000 7fa39a3aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:03.180 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:03.174+0000 7fa39a3aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:03.180 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:03.174+0000 7fa39a3aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:03.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:26:03.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:26:03.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:26:03.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:26:03.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:26:03.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:26:03.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:26:03.314 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:26:03.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:26:03.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:26:03.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:26:03.587 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:03.582+0000 7fa39a3aba40 -1 Falling back to public interface 2026-02-22T20:26:04.225 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:04.218+0000 7fa39a3aba40 -1 osd.0 19 log_to_monitors true 2026-02-22T20:26:04.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:26:04.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:26:04.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:26:04.473 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:26:04.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:26:04.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:26:04.629 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 20 last_clean_interval [5,19) [v2:127.0.0.1:6802/2700788302,v1:127.0.0.1:6803/2700788302] [v2:127.0.0.1:6804/2700788302,v1:127.0.0.1:6805/2700788302] exists,up 318617e4-01ea-4819-ace4-a5a4256a74b1 2026-02-22T20:26:04.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:26:04.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:26:04.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:26:04.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:26:04.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:26:04.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:26:04.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:26:04.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:26:04.631 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:26:04.631 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:26:04.631 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:26:04.631 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:26:04.631 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:26:04.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:26:04.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:26:04.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:26:04.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:26:04.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:26:04.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:26:04.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:26:04.897 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:26:04.898 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:26:04.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:26:04.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:04.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:26:04.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-22T20:26:04.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-22T20:26:04.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-22T20:26:04.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:04.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:26:05.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-22T20:26:05.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-22T20:26:05.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-42949672965' 2026-02-22T20:26:05.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:05.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:26:05.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-22T20:26:05.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-22T20:26:05.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-42949672965 2-60129542148' 2026-02-22T20:26:05.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:05.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-22T20:26:05.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:05.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:26:05.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-22T20:26:05.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:05.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-22T20:26:05.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-22T20:26:05.118 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 98784247810 2026-02-22T20:26:05.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:05.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 98784247810 2026-02-22T20:26:05.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:26:06.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:26:06.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:06.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247810 2026-02-22T20:26:06.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:06.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-22T20:26:06.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:06.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:26:06.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-22T20:26:06.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:06.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-22T20:26:06.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-22T20:26:06.445 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672965 2026-02-22T20:26:06.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:26:06.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-22T20:26:06.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:06.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-22T20:26:06.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:06.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:26:06.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-22T20:26:06.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:06.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-22T20:26:06.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-22T20:26:06.619 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542148 2026-02-22T20:26:06.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:26:06.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-22T20:26:06.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:26:06.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:06.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:06.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:26:06.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:26:06.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:26:06.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:26:06.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:26:06.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:26:06.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:26:06.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:26:07.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:26:07.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:26:07.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:07.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:07.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:26:07.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:26:07.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:26:07.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_scrub: get_pg testpool SOMETHING 2026-02-22T20:26:07.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-22T20:26:07.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:26:07.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2026-02-22T20:26:07.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:26:07.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:569: TEST_auto_repair_bluestore_scrub: local pgid=1.0 2026-02-22T20:26:07.466 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:570: TEST_auto_repair_bluestore_scrub: get_primary testpool SOMETHING 2026-02-22T20:26:07.466 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:26:07.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:26:07.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:26:07.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:26:07.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:570: TEST_auto_repair_bluestore_scrub: local primary=1 2026-02-22T20:26:07.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_scrub: get_last_scrub_stamp 1.0 2026-02-22T20:26:07.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:26:07.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:26:07.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:26:07.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:26:07.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:571: TEST_auto_repair_bluestore_scrub: local last_scrub_stamp=2026-02-22T20:25:57.120326+0000 2026-02-22T20:26:07.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:572: TEST_auto_repair_bluestore_scrub: ceph tell 1.0 schedule-scrub 2026-02-22T20:26:07.843 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:26:07.843 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:26:07.844 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:26:07.844 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-20T20:26:07.841234+0000" 2026-02-22T20:26:07.844 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:26:07.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:575: TEST_auto_repair_bluestore_scrub: wait_for_scrub 1.0 2026-02-22T20:25:57.120326+0000 2026-02-22T20:26:07.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:26:07.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:25:57.120326+0000 2026-02-22T20:26:07.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:26:07.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:26:07.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:26:07.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:26:07.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:26:07.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:26:07.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:26:07.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:26:08.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:25:57.120326+0000 '>' 2026-02-22T20:25:57.120326+0000 2026-02-22T20:26:08.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:26:09.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:26:09.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:26:09.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:26:09.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:26:09.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:26:09.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:26:09.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:26:09.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:25:57.120326+0000 '>' 2026-02-22T20:25:57.120326+0000 2026-02-22T20:26:09.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:26:10.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:26:10.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:26:10.175 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:26:10.175 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:26:10.175 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:26:10.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:26:10.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:26:10.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:25:57.120326+0000 '>' 2026-02-22T20:25:57.120326+0000 2026-02-22T20:26:10.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:26:11.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:26:11.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:26:11.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:26:11.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:26:11.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:26:11.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:26:11.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:26:11.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:25:57.120326+0000 '>' 2026-02-22T20:25:57.120326+0000 2026-02-22T20:26:11.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:26:12.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:26:12.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:26:12.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:26:12.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:26:12.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:26:12.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:26:12.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:26:12.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:25:57.120326+0000 '>' 2026-02-22T20:25:57.120326+0000 2026-02-22T20:26:12.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:26:13.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:26:13.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:26:13.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:26:13.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:26:13.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:26:13.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:26:13.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:26:13.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:25:57.120326+0000 '>' 2026-02-22T20:25:57.120326+0000 2026-02-22T20:26:13.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:26:14.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:26:14.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:26:14.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:26:14.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:26:14.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:26:14.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:26:14.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:26:14.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:26:09.717184+0000 '>' 2026-02-22T20:25:57.120326+0000 2026-02-22T20:26:14.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:26:14.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:576: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2026-02-22T20:26:15.128 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:26:15.128 INFO:tasks.workunit.client.0.trial150.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-22T20:26:09.719592+0000 19'1 24:62 [1,0] 1 [1,0] 1 19'1 2026-02-22T20:26:09.717184+0000 19'1 2026-02-22T20:26:09.717184+0000 0 1 periodic scrub scheduled @ 2026-02-23T20:26:09.717184+0000 1 0 2026-02-22T20:26:15.128 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:26:15.128 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:26:15.128 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:26:15.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:578: TEST_auto_repair_bluestore_scrub: sleep 2 2026-02-22T20:26:17.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:579: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2026-02-22T20:26:17.285 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:26:17.286 INFO:tasks.workunit.client.0.trial150.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-22T20:26:09.719592+0000 19'1 24:62 [1,0] 1 [1,0] 1 19'1 2026-02-22T20:26:09.717184+0000 19'1 2026-02-22T20:26:09.717184+0000 0 1 periodic scrub scheduled @ 2026-02-23T20:26:09.717184+0000 1 0 2026-02-22T20:26:17.286 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:26:17.286 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:26:17.287 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:26:17.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:580: TEST_auto_repair_bluestore_scrub: sleep 2 2026-02-22T20:26:19.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:581: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2026-02-22T20:26:19.444 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:26:19.444 INFO:tasks.workunit.client.0.trial150.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-22T20:26:09.719592+0000 19'1 24:62 [1,0] 1 [1,0] 1 19'1 2026-02-22T20:26:09.717184+0000 19'1 2026-02-22T20:26:09.717184+0000 0 1 periodic scrub scheduled @ 2026-02-23T20:26:09.717184+0000 1 0 2026-02-22T20:26:19.444 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:26:19.444 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:26:19.444 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:26:19.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:582: TEST_auto_repair_bluestore_scrub: sleep 5 2026-02-22T20:26:24.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:583: TEST_auto_repair_bluestore_scrub: wait_for_clean 2026-02-22T20:26:24.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:26:24.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:26:24.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:26:24.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:26:24.455 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:26:24.455 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:26:24.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:26:24.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:26:24.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:26:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:26:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:26:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:26:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:26:24.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:26:24.566 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:26:24.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:26:24.729 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:26:24.729 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:26:24.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:26:24.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:24.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:26:24.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247815 2026-02-22T20:26:24.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247815 2026-02-22T20:26:24.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247815' 2026-02-22T20:26:24.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:24.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:26:24.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672970 2026-02-22T20:26:24.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672970 2026-02-22T20:26:24.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247815 1-42949672970' 2026-02-22T20:26:24.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:24.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:26:24.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-22T20:26:24.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-22T20:26:24.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247815 1-42949672970 2-60129542153' 2026-02-22T20:26:24.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:24.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247815 2026-02-22T20:26:24.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:24.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:26:24.951 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247815 2026-02-22T20:26:24.951 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:24.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247815 2026-02-22T20:26:24.952 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 98784247815 2026-02-22T20:26:24.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247815' 2026-02-22T20:26:24.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:25.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247813 -lt 98784247815 2026-02-22T20:26:25.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:26:26.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:26:26.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:26.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247813 -lt 98784247815 2026-02-22T20:26:26.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:26:27.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:26:27.280 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:27.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247815 -lt 98784247815 2026-02-22T20:26:27.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:27.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672970 2026-02-22T20:26:27.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:27.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:26:27.449 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672970 2026-02-22T20:26:27.449 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:27.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672970 2026-02-22T20:26:27.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2026-02-22T20:26:27.451 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672970 2026-02-22T20:26:27.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:26:27.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672970 -lt 42949672970 2026-02-22T20:26:27.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:27.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-22T20:26:27.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:27.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:26:27.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-22T20:26:27.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:27.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-22T20:26:27.624 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542153 2026-02-22T20:26:27.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-22T20:26:27.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:26:27.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542153 2026-02-22T20:26:27.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:26:27.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:27.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:27.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:26:27.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:26:27.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:26:27.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:26:27.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:26:27.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:26:27.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:26:27.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:26:28.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:26:28.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:26:28.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:28.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:28.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:26:28.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:26:28.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:26:28.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:584: TEST_auto_repair_bluestore_scrub: ceph pg dump pgs 2026-02-22T20:26:28.488 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:26:28.488 INFO:tasks.workunit.client.0.trial150.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean 2026-02-22T20:26:09.719592+0000 19'1 24:62 [1,0] 1 [1,0] 1 19'1 2026-02-22T20:26:09.717184+0000 19'1 2026-02-22T20:26:09.717184+0000 0 1 periodic scrub scheduled @ 2026-02-23T20:26:09.717184+0000 1 0 2026-02-22T20:26:28.488 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:26:28.489 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:26:28.489 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:26:28.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_scrub: get_not_primary testpool SOMETHING 2026-02-22T20:26:28.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:26:28.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:26:28.500 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-22T20:26:28.500 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:26:28.500 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:26:28.501 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:26:28.501 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:26:28.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:26:28.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:26:28.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:26:28.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:587: TEST_auto_repair_bluestore_scrub: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:26:28.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:26:28.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:26:28.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:26:28.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:26:28.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:26:28.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:26:28.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:26:28.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:26:28.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:26:28.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:26:28.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:26:28.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:26:28.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:26:28.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:26:28.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:26:29.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:26:29.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:26:29.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:26:29.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:26:29.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:26:29.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:26:29.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:26:29.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-22T20:26:29.209 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:26:29.209 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:26:29.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-22T20:26:29.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:26:29.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:26:29.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:26:29.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:26:29.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:26:29.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:26:29.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:26:29.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:26:29.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:26:29.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:26:29.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:26:29.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:26:29.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:26:29.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:26:29.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:26:29.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:26:29.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:26:29.309 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:26:29.310 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:29.310 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:29.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:29.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:29.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:26:29.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:26:29.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:26:29.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:26:29.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:26:29.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:26:29.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:26:29.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:26:29.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0' 2026-02-22T20:26:29.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:26:29.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:26:29.312 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:26:29.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-scrub-backoff-ratio=0 2026-02-22T20:26:29.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:26:29.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:26:29.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:26:29.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:26:29.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:26:29.332 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:29.326+0000 7f52d4c50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:29.334 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:29.330+0000 7f52d4c50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:29.335 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:29.330+0000 7f52d4c50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:29.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:26:29.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:26:29.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:26:29.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:26:29.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:26:29.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:26:29.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:26:29.467 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:26:29.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:26:29.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:26:29.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:26:29.711 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:29.706+0000 7f52d4c50a40 -1 Falling back to public interface 2026-02-22T20:26:30.331 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:30.326+0000 7f52d4c50a40 -1 osd.0 24 log_to_monitors true 2026-02-22T20:26:30.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:26:30.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:26:30.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:26:30.625 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:26:30.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:26:30.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:26:30.782 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 28 up_thru 0 down_at 25 last_clean_interval [23,24) [v2:127.0.0.1:6802/3197930440,v1:127.0.0.1:6803/3197930440] [v2:127.0.0.1:6804/3197930440,v1:127.0.0.1:6805/3197930440] exists,up 318617e4-01ea-4819-ace4-a5a4256a74b1 2026-02-22T20:26:30.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:26:30.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:26:30.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:26:30.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:26:30.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:26:30.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:26:30.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:26:30.782 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:26:30.783 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:26:30.783 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:26:30.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:26:30.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:26:30.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:26:30.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:26:30.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:26:30.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:26:30.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:26:30.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:26:30.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:26:31.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:26:31.041 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:26:31.041 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:26:31.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:26:31.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:31.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:26:31.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084290 2026-02-22T20:26:31.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084290 2026-02-22T20:26:31.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290' 2026-02-22T20:26:31.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:31.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:26:31.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672972 2026-02-22T20:26:31.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672972 2026-02-22T20:26:31.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290 1-42949672972' 2026-02-22T20:26:31.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:31.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:26:31.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-22T20:26:31.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-22T20:26:31.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-120259084290 1-42949672972 2-60129542155' 2026-02-22T20:26:31.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:31.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-120259084290 2026-02-22T20:26:31.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:31.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:26:31.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-120259084290 2026-02-22T20:26:31.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:31.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084290 2026-02-22T20:26:31.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 120259084290' 2026-02-22T20:26:31.269 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 120259084290 2026-02-22T20:26:31.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:31.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247815 -lt 120259084290 2026-02-22T20:26:31.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:26:32.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:26:32.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:32.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084290 -lt 120259084290 2026-02-22T20:26:32.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:32.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672972 2026-02-22T20:26:32.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:32.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:26:32.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672972 2026-02-22T20:26:32.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:32.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672972 2026-02-22T20:26:32.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2026-02-22T20:26:32.590 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672972 2026-02-22T20:26:32.590 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:26:32.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672972 -lt 42949672972 2026-02-22T20:26:32.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:32.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-22T20:26:32.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:32.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:26:32.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-22T20:26:32.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:32.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-22T20:26:32.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-22T20:26:32.755 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542155 2026-02-22T20:26:32.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:26:32.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542156 -lt 60129542155 2026-02-22T20:26:32.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:26:32.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:32.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:33.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:26:33.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:26:33.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:26:33.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:26:33.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:26:33.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:26:33.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:26:33.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:26:33.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:26:33.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:26:33.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:33.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:33.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:26:33.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:26:33.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:26:33.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:588: TEST_auto_repair_bluestore_scrub: rados --pool testpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:26:33.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:589: TEST_auto_repair_bluestore_scrub: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:26:33.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:590: TEST_auto_repair_bluestore_scrub: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-22T20:26:33.485 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:26:08.690+0000 7f833272a640 10 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=15/15 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing(periodic-regular,deep-scrub-on-error)] scrubber: scrub_finish before flags: DEEP_SCRUB_ON_ERROR. repair state: no-repair. deep_scrub_on_error: 1 2026-02-22T20:26:33.485 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:26:08.690+0000 7f833272a640 10 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=15/15 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing(periodic-regular,deep-scrub-on-error)] scrubber: scrub_finish: will initiate a deep scrub to fix 1 errors 2026-02-22T20:26:33.485 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:26:08.690+0000 7f833272a640 10 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=15/15 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing(periodic-regular)] scrubber: _scrub_finish info stats: valid m_is_repair: 0 2026-02-22T20:26:33.485 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:26:09.714+0000 7f833272a640 10 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=15/15 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+repair(repairing,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: repair. deep_scrub_on_error: 0 2026-02-22T20:26:33.485 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:26:09.714+0000 7f833272a640 10 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=15/15 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep+inconsistent+repair(repairing,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-22T20:26:33.485 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:26:09.714+0000 7f833272a640 15 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=15/15 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair(repairing,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2026-02-22T20:26:33.485 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:26:09.714+0000 7f833272a640 20 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=15/15 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair(repairing,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2026-02-22T20:26:33.485 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:26:09.714+0000 7f833272a640 19 osd.1 pg_epoch: 24 pg[1.0( v 19'1 (0'0,19'1] local-lis/les=23/24 n=1 ec=15/15 lis/c=23/23 les/c/f=24/24/0 sis=23) [1,0] r=0 lpr=23 crt=19'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+inconsistent+repair(repairing,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-22T20:26:33.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_scrub: ceph pg 1.0 query 2026-02-22T20:26:33.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_scrub: jq .info.stats.stat_sum.num_objects_repaired 2026-02-22T20:26:33.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:593: TEST_auto_repair_bluestore_scrub: COUNT=1 2026-02-22T20:26:33.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:594: TEST_auto_repair_bluestore_scrub: test 1 = 1 2026-02-22T20:26:33.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:26:33.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:26:33.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:26:33.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:26:33.560 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:26:33.560 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:26:33.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:26:33.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:26:33.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:26:33.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:26:33.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:26:33.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:26:33.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:26:33.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:26:33.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:26:33.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:26:33.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:26:33.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:26:33.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:26:33.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:26:33.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:26:33.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:26:33.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:26:33.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:26:33.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:33.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:33.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:26:33.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:26:33.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:26:33.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:26:33.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:26:33.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:26:33.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:26:33.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:26:33.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:26:33.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:26:33.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:26:33.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:26:33.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:26:33.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:26:33.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:26:33.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:26:33.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:26:33.704 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:26:33.704 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:33.704 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:33.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:26:33.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:26:33.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:26:33.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:26:33.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:26:33.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:33.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:33.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:26:33.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_bluestore_tag td/osd-scrub-repair 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:443: TEST_auto_repair_bluestore_tag: local dir=td/osd-scrub-repair 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:444: TEST_auto_repair_bluestore_tag: local poolname=testpool 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:447: TEST_auto_repair_bluestore_tag: run_mon td/osd-scrub-repair a 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:26:33.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:26:33.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:26:33.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:26:33.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:26:33.732 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:26:33.732 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:33.732 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:33.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:33.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:26:33.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:26:33.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:26:33.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:26:33.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:26:33.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:26:33.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:26:33.765 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:26:33.765 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:26:33.765 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:26:33.766 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:26:33.766 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:33.766 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:33.766 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:26:33.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:26:33.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:26:33.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:26:33.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:26:33.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:26:33.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:26:33.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:26:33.824 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:26:33.824 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:26:33.824 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:26:33.824 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:26:33.825 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:33.825 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:33.825 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:26:33.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:26:33.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:26:33.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:448: TEST_auto_repair_bluestore_tag: run_mgr td/osd-scrub-repair x 2026-02-22T20:26:33.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:26:33.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:26:33.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:26:33.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:26:33.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:26:33.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:26:33.976 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:26:33.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:26:33.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:26:33.977 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:26:33.977 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:33.977 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:33.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:33.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:26:33.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:26:33.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:455: TEST_auto_repair_bluestore_tag: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-22T20:26:33.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_tag: seq 0 2 2026-02-22T20:26:34.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2026-02-22T20:26:34.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:457: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-22T20:26:34.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:26:34.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:26:34.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:26:34.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:26:34.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:26:34.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:26:34.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:26:34.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:26:34.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:26:34.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:26:34.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:26:34.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:26:34.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:26:34.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:26:34.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:26:34.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:26:34.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:26:34.002 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:26:34.002 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:34.002 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:34.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:34.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:34.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:26:34.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:26:34.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:26:34.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:26:34.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:26:34.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:26:34.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:26:34.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:26:34.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:26:34.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:26:34.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-22T20:26:34.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:26:34.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:26:34.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=242d2df6-2d7b-4b4a-a2b3-88106e2b9404 2026-02-22T20:26:34.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 242d2df6-2d7b-4b4a-a2b3-88106e2b9404' 2026-02-22T20:26:34.005 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 242d2df6-2d7b-4b4a-a2b3-88106e2b9404 2026-02-22T20:26:34.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:26:34.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB6Zptpqd+4ABAAH+uacADWdNsAyNB53kM56w== 2026-02-22T20:26:34.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB6Zptpqd+4ABAAH+uacADWdNsAyNB53kM56w=="}' 2026-02-22T20:26:34.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 242d2df6-2d7b-4b4a-a2b3-88106e2b9404 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:26:34.104 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:26:34.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:26:34.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQB6Zptpqd+4ABAAH+uacADWdNsAyNB53kM56w== --osd-uuid 242d2df6-2d7b-4b4a-a2b3-88106e2b9404 2026-02-22T20:26:34.131 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:34.126+0000 7f7a7750ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:34.133 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:34.126+0000 7f7a7750ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:34.133 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:34.130+0000 7f7a7750ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:34.134 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:34.130+0000 7f7a7750ba40 -1 bdev(0x5564f8b04c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:26:34.134 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:34.130+0000 7f7a7750ba40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:26:34.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:26:34.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:26:34.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:26:34.860 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:26:34.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:26:35.061 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:26:35.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:26:35.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-22T20:26:35.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:26:35.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:26:35.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:26:35.089 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:35.082+0000 7ff9b3d2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:35.091 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:35.086+0000 7ff9b3d2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:35.092 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:35.086+0000 7ff9b3d2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:35.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:26:35.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:26:35.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:26:35.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:26:35.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:26:35.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:26:35.226 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:26:35.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:26:35.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:26:35.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:26:35.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:26:35.515 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:35.510+0000 7ff9b3d2aa40 -1 Falling back to public interface 2026-02-22T20:26:36.172 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:36.166+0000 7ff9b3d2aa40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:26:36.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:26:36.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:26:36.385 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:26:36.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:26:36.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:26:36.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:26:36.545 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2234923143,v1:127.0.0.1:6803/2234923143] [v2:127.0.0.1:6804/2234923143,v1:127.0.0.1:6805/2234923143] exists,up 242d2df6-2d7b-4b4a-a2b3-88106e2b9404 2026-02-22T20:26:36.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:26:36.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:26:36.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:26:36.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2026-02-22T20:26:36.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:457: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-22T20:26:36.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:26:36.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:26:36.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:26:36.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:26:36.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:26:36.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:26:36.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:26:36.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:26:36.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:26:36.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:26:36.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:26:36.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:26:36.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:26:36.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:26:36.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:26:36.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:26:36.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:26:36.550 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:26:36.550 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:36.550 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:36.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:36.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:36.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:26:36.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:26:36.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:26:36.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:26:36.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:26:36.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:26:36.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:26:36.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:26:36.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:26:36.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:26:36.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-22T20:26:36.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:26:36.552 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:26:36.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7ae39dca-73f1-4076-9521-a134573404fa 2026-02-22T20:26:36.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7ae39dca-73f1-4076-9521-a134573404fa' 2026-02-22T20:26:36.553 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 7ae39dca-73f1-4076-9521-a134573404fa 2026-02-22T20:26:36.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:26:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB8ZptpGUZoIRAAj2x0ZZ9JKkncjWcBRPaMJQ== 2026-02-22T20:26:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB8ZptpGUZoIRAAj2x0ZZ9JKkncjWcBRPaMJQ=="}' 2026-02-22T20:26:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7ae39dca-73f1-4076-9521-a134573404fa -i td/osd-scrub-repair/1/new.json 2026-02-22T20:26:36.727 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:26:36.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:26:36.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQB8ZptpGUZoIRAAj2x0ZZ9JKkncjWcBRPaMJQ== --osd-uuid 7ae39dca-73f1-4076-9521-a134573404fa 2026-02-22T20:26:36.755 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:36.750+0000 7fc5122a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:36.757 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:36.750+0000 7fc5122a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:36.758 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:36.754+0000 7fc5122a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:36.759 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:36.754+0000 7fc5122a5a40 -1 bdev(0x55f606d6cc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:26:36.759 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:36.754+0000 7fc5122a5a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:26:37.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:26:37.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:26:37.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:26:37.386 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:26:37.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:26:37.587 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:26:37.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:26:37.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-22T20:26:37.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:26:37.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:26:37.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:26:37.613 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:37.606+0000 7f9ca5766a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:37.615 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:37.610+0000 7f9ca5766a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:37.616 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:37.610+0000 7f9ca5766a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:37.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:26:37.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:26:37.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:26:37.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:26:37.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:26:37.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:26:37.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:26:37.748 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:26:37.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:26:37.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:26:37.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:26:37.979 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:37.974+0000 7f9ca5766a40 -1 Falling back to public interface 2026-02-22T20:26:38.571 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:38.566+0000 7f9ca5766a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:26:38.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:26:38.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:26:38.910 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:26:38.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:26:38.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:26:38.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:26:39.068 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/635959446,v1:127.0.0.1:6811/635959446] [v2:127.0.0.1:6812/635959446,v1:127.0.0.1:6813/635959446] exists,up 7ae39dca-73f1-4076-9521-a134573404fa 2026-02-22T20:26:39.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:26:39.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:26:39.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:26:39.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:456: TEST_auto_repair_bluestore_tag: for id in $(seq 0 2) 2026-02-22T20:26:39.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:457: TEST_auto_repair_bluestore_tag: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-22T20:26:39.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:26:39.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:26:39.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:26:39.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:26:39.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:26:39.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:26:39.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:26:39.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:26:39.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:26:39.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:26:39.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:26:39.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:26:39.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:26:39.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:26:39.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:26:39.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:26:39.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:26:39.075 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:26:39.075 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:39.075 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:39.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:39.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:39.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:26:39.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:26:39.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:26:39.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:26:39.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:26:39.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:26:39.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:26:39.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:26:39.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:26:39.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:26:39.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-22T20:26:39.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:26:39.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:26:39.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=42be61f5-627f-4045-b344-d94c1cf61b66 2026-02-22T20:26:39.078 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 42be61f5-627f-4045-b344-d94c1cf61b66 2026-02-22T20:26:39.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 42be61f5-627f-4045-b344-d94c1cf61b66' 2026-02-22T20:26:39.078 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:26:39.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB/ZptpVB3mBBAAHKDrMO690fUupuPLxjA6Qg== 2026-02-22T20:26:39.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB/ZptpVB3mBBAAHKDrMO690fUupuPLxjA6Qg=="}' 2026-02-22T20:26:39.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 42be61f5-627f-4045-b344-d94c1cf61b66 -i td/osd-scrub-repair/2/new.json 2026-02-22T20:26:39.245 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:26:39.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:26:39.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq --mkfs --key AQB/ZptpVB3mBBAAHKDrMO690fUupuPLxjA6Qg== --osd-uuid 42be61f5-627f-4045-b344-d94c1cf61b66 2026-02-22T20:26:39.272 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:39.266+0000 7f65a0affa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:39.274 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:39.270+0000 7f65a0affa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:39.275 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:39.270+0000 7f65a0affa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:39.275 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:39.270+0000 7f65a0affa40 -1 bdev(0x55ad75e90c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:26:39.275 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:39.270+0000 7f65a0affa40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:26:39.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:26:39.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:26:39.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:26:39.946 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:26:39.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:26:40.151 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:26:40.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:26:40.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-22T20:26:40.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:26:40.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:26:40.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:26:40.177 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:40.170+0000 7f53fe76ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:40.178 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:40.174+0000 7f53fe76ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:40.179 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:40.174+0000 7f53fe76ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:40.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:26:40.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:26:40.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:26:40.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:26:40.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:26:40.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:26:40.314 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:26:40.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:26:40.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:26:40.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:26:40.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:26:40.571 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:40.566+0000 7f53fe76ca40 -1 Falling back to public interface 2026-02-22T20:26:41.134 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:41.130+0000 7f53fe76ca40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:26:41.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:26:41.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:26:41.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:26:41.475 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:26:41.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:26:41.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:26:41.633 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3798740885,v1:127.0.0.1:6819/3798740885] [v2:127.0.0.1:6820/3798740885,v1:127.0.0.1:6821/3798740885] exists,up 42be61f5-627f-4045-b344-d94c1cf61b66 2026-02-22T20:26:41.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:26:41.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:26:41.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:26:41.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:460: TEST_auto_repair_bluestore_tag: create_pool testpool 1 1 2026-02-22T20:26:41.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-22T20:26:41.825 INFO:tasks.workunit.client.0.trial150.stderr:pool 'testpool' created 2026-02-22T20:26:41.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:26:42.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:461: TEST_auto_repair_bluestore_tag: ceph osd pool set testpool size 2 2026-02-22T20:26:43.038 INFO:tasks.workunit.client.0.trial150.stderr:set pool 1 size to 2 2026-02-22T20:26:43.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:462: TEST_auto_repair_bluestore_tag: wait_for_clean 2026-02-22T20:26:43.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:26:43.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:26:43.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:26:43.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:26:43.051 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:26:43.051 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:26:43.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:26:43.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:26:43.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:26:43.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:26:43.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:26:43.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:26:43.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:26:43.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:26:43.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:26:43.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:26:43.317 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:26:43.317 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:26:43.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:26:43.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:43.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:26:43.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:26:43.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:26:43.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:26:43.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:43.388 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:26:43.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672962 2026-02-22T20:26:43.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672962 2026-02-22T20:26:43.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672962' 2026-02-22T20:26:43.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:43.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:26:43.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509442 2026-02-22T20:26:43.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509442 2026-02-22T20:26:43.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672962 2-64424509442' 2026-02-22T20:26:43.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:43.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:26:43.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:43.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:26:43.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:26:43.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:43.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:26:43.536 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:26:43.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:26:43.537 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:43.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:26:43.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:26:44.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:26:44.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:44.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T20:26:44.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:44.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672962 2026-02-22T20:26:44.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:44.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:26:44.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672962 2026-02-22T20:26:44.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:44.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672962 2026-02-22T20:26:44.863 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672962 2026-02-22T20:26:44.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2026-02-22T20:26:44.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:26:45.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672962 -lt 42949672962 2026-02-22T20:26:45.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:45.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509442 2026-02-22T20:26:45.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:45.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:26:45.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509442 2026-02-22T20:26:45.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:45.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509442 2026-02-22T20:26:45.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2026-02-22T20:26:45.036 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509442 2026-02-22T20:26:45.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:26:45.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509442 -lt 64424509442 2026-02-22T20:26:45.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:26:45.213 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:45.213 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:45.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:26:45.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:26:45.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:26:45.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:26:45.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:26:45.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:26:45.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:26:45.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:26:45.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:26:45.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:26:45.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:45.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:45.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:26:45.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-22T20:26:45.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:26:45.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-22T20:26:45.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:26:45.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:26:45.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:26:45.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:26:45.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:26:45.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:26:45.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:26:45.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:26:45.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:26:45.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:26:46.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:26:46.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:26:46.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:46.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:46.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:26:46.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:26:46.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:26:46.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:26:46.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:26:46.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:26:46.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:26:46.200 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:26:46.200 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:26:46.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:26:46.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:26:46.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:26:46.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:26:46.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:26:46.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:26:46.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:26:46.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:26:46.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:26:46.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:26:46.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:26:46.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:26:46.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:26:46.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:26:46.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:26:46.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:46.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:46.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:26:46.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:26:46.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:26:46.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:465: TEST_auto_repair_bluestore_tag: local payload=ABCDEF 2026-02-22T20:26:46.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:466: TEST_auto_repair_bluestore_tag: echo ABCDEF 2026-02-22T20:26:46.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:467: TEST_auto_repair_bluestore_tag: rados --pool testpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:26:46.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2026-02-22T20:26:46.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:26:46.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:26:46.952 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-22T20:26:46.952 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:26:46.952 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:26:46.952 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:26:46.952 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:26:47.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:26:47.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:26:47.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:26:47.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:471: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:26:47.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:26:47.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:26:47.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:26:47.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:26:47.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:26:47.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:26:47.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:26:47.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:26:47.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:26:47.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:26:47.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:26:47.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:26:47.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:26:47.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:26:47.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:26:47.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:26:47.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:26:47.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:26:47.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:26:47.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:26:47.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:26:47.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:26:47.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-22T20:26:47.739 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-22T20:26:47.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-22T20:26:47.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:26:47.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-22T20:26:47.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:26:47.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:26:47.850 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:26:47.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-22T20:26:47.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:26:47.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:26:47.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:26:47.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:26:47.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:26:47.870 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:47.866+0000 7f9b7087aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:47.872 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:47.866+0000 7f9b7087aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:47.873 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:47.866+0000 7f9b7087aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:26:48.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:26:48.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:26:48.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:26:48.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:26:48.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:26:48.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:26:48.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:26:48.006 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:26:48.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:26:48.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:26:48.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:26:48.303 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:48.298+0000 7f9b7087aa40 -1 Falling back to public interface 2026-02-22T20:26:48.942 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:26:48.938+0000 7f9b7087aa40 -1 osd.0 20 log_to_monitors true 2026-02-22T20:26:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:26:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:26:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:26:49.170 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:26:49.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:26:49.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:26:49.330 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6802/2591127189,v1:127.0.0.1:6803/2591127189] [v2:127.0.0.1:6804/2591127189,v1:127.0.0.1:6805/2591127189] exists,up 242d2df6-2d7b-4b4a-a2b3-88106e2b9404 2026-02-22T20:26:49.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:26:49.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:26:49.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:26:49.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:26:49.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:26:49.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:26:49.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:26:49.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:26:49.331 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:26:49.331 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:26:49.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:26:49.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:26:49.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:26:49.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:26:49.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:26:49.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:26:49.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:26:49.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:26:49.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:26:49.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:26:49.596 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:26:49.596 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:26:49.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:26:49.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:49.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:26:49.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-22T20:26:49.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-22T20:26:49.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-22T20:26:49.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:49.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:26:49.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-22T20:26:49.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-22T20:26:49.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-42949672965' 2026-02-22T20:26:49.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:49.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:26:49.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509444 2026-02-22T20:26:49.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509444 2026-02-22T20:26:49.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-42949672965 2-64424509444' 2026-02-22T20:26:49.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:49.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-22T20:26:49.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:49.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:26:49.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-22T20:26:49.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:49.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-22T20:26:49.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-22T20:26:49.816 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 103079215106 2026-02-22T20:26:49.816 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:49.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 103079215106 2026-02-22T20:26:49.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:26:50.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:26:50.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:51.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215106 2026-02-22T20:26:51.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:51.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-22T20:26:51.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:51.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:26:51.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-22T20:26:51.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:51.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-22T20:26:51.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-22T20:26:51.136 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672965 2026-02-22T20:26:51.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:26:51.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-22T20:26:51.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:51.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509444 2026-02-22T20:26:51.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:51.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:26:51.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509444 2026-02-22T20:26:51.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:51.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509444 2026-02-22T20:26:51.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2026-02-22T20:26:51.298 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509444 2026-02-22T20:26:51.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:26:51.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509444 -lt 64424509444 2026-02-22T20:26:51.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:26:51.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:51.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:51.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:26:51.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:26:51.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:26:51.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:26:51.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:26:51.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:26:51.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:26:51.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:26:51.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:26:51.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:26:51.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:26:51.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:26:52.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:26:52.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:26:52.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:26:52.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_tag: get_pg testpool SOMETHING 2026-02-22T20:26:52.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-22T20:26:52.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:26:52.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool SOMETHING 2026-02-22T20:26:52.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:26:52.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:473: TEST_auto_repair_bluestore_tag: local pgid=1.0 2026-02-22T20:26:52.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:474: TEST_auto_repair_bluestore_tag: get_primary testpool SOMETHING 2026-02-22T20:26:52.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:26:52.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:26:52.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:26:52.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:26:52.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:474: TEST_auto_repair_bluestore_tag: local primary=1 2026-02-22T20:26:52.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:475: TEST_auto_repair_bluestore_tag: echo 'Affected PG ' 1.0 ' w/ primary ' 1 2026-02-22T20:26:52.337 INFO:tasks.workunit.client.0.trial150.stdout:Affected PG 1.0 w/ primary 1 2026-02-22T20:26:52.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_tag: get_last_scrub_stamp 1.0 2026-02-22T20:26:52.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:26:52.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:26:52.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:26:52.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:26:52.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:476: TEST_auto_repair_bluestore_tag: local last_scrub_stamp=2026-02-22T20:26:41.821166+0000 2026-02-22T20:26:52.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:477: TEST_auto_repair_bluestore_tag: initiate_and_fetch_state 1 1.0 3.0 2026-02-22T20:26:52.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:382: initiate_and_fetch_state: local the_osd=osd.1 2026-02-22T20:26:52.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:383: initiate_and_fetch_state: local pgid=1.0 2026-02-22T20:26:52.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: initiate_and_fetch_state: get_last_scrub_stamp 1.0 2026-02-22T20:26:52.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:26:52.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:26:52.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:26:52.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:26:52.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:384: initiate_and_fetch_state: local last_scrub=2026-02-22T20:26:41.821166+0000 2026-02-22T20:26:52.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:386: initiate_and_fetch_state: set_config osd 1 osd_scrub_sleep 3.0 2026-02-22T20:26:52.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:26:52.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-22T20:26:52.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2026-02-22T20:26:52.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=3.0 2026-02-22T20:26:52.651 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:26:52.652 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-22T20:26:52.653 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:26:52.653 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:26:52.653 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:26:52.653 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:52.653 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:52.653 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:26:52.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.1.asok config set osd_scrub_sleep 3.0 2026-02-22T20:26:52.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:26:52.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:387: initiate_and_fetch_state: set_config osd 1 osd_scrub_auto_repair true 2026-02-22T20:26:52.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:26:52.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-22T20:26:52.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_auto_repair 2026-02-22T20:26:52.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:26:52.714 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:26:52.715 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-22T20:26:52.715 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:26:52.716 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:26:52.716 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:26:52.716 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:52.716 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:52.716 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:26:52.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.1.asok config set osd_scrub_auto_repair true 2026-02-22T20:26:52.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:26:52.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:389: initiate_and_fetch_state: flush_pg_stats 2026-02-22T20:26:52.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:26:52.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:26:52.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:26:52.938 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:26:52.938 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:26:52.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:26:52.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:52.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:26:53.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215107 2026-02-22T20:26:53.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215107 2026-02-22T20:26:53.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215107' 2026-02-22T20:26:53.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:53.013 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:26:53.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672966 2026-02-22T20:26:53.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672966 2026-02-22T20:26:53.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215107 1-42949672966' 2026-02-22T20:26:53.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:26:53.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:26:53.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509446 2026-02-22T20:26:53.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509446 2026-02-22T20:26:53.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215107 1-42949672966 2-64424509446' 2026-02-22T20:26:53.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:53.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215107 2026-02-22T20:26:53.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:53.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:26:53.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215107 2026-02-22T20:26:53.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:53.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215107 2026-02-22T20:26:53.161 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 103079215107 2026-02-22T20:26:53.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215107' 2026-02-22T20:26:53.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:53.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215107 2026-02-22T20:26:53.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:26:54.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:26:54.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:54.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215107 2026-02-22T20:26:54.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:26:55.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:26:55.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:26:55.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215108 -lt 103079215107 2026-02-22T20:26:55.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:55.652 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672966 2026-02-22T20:26:55.652 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:55.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:26:55.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672966 2026-02-22T20:26:55.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:55.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672966 2026-02-22T20:26:55.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2026-02-22T20:26:55.656 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672966 2026-02-22T20:26:55.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:26:55.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672966 -lt 42949672966 2026-02-22T20:26:55.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:26:55.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509446 2026-02-22T20:26:55.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:26:55.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:26:55.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509446 2026-02-22T20:26:55.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:26:55.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509446 2026-02-22T20:26:55.828 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509446 2026-02-22T20:26:55.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2026-02-22T20:26:55.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:26:55.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509446 -lt 64424509446 2026-02-22T20:26:55.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:390: initiate_and_fetch_state: date --rfc-3339=ns 2026-02-22T20:26:55.999 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22 20:26:55.997463752+00:00 2026-02-22T20:26:55.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:393: initiate_and_fetch_state: get_asok_path osd.1 2026-02-22T20:26:56.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:26:56.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:26:56.000 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:26:56.001 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:26:56.001 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:26:56.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:26:56.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:393: initiate_and_fetch_state: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.1.asok schedule-deep-scrub 1.0 2026-02-22T20:26:56.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i=0 )) 2026-02-22T20:26:56.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:26:56.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:26:56.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:26:56.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:26:56.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 0 ') state now: ' '"active+clean"' 2026-02-22T20:26:56.143 INFO:tasks.workunit.client.0.trial150.stdout:{"deep":true,"must":false,"stamp":"2026-02-11T15:38:46.058440+0000"}0 ) state now: "active+clean" 2026-02-22T20:26:56.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:26:56.143 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:26:56.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:26:56.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 0 == 4 ']' 2026-02-22T20:26:56.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:26:56.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:26:56.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:26:56.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:26:56.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:26:56.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:26:56.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 1 ') state now: ' '"active+clean"' 2026-02-22T20:26:56.519 INFO:tasks.workunit.client.0.trial150.stdout:1 ) state now: "active+clean" 2026-02-22T20:26:56.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:26:56.520 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:26:56.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:26:56.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 1 == 4 ']' 2026-02-22T20:26:56.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:26:56.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:26:56.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:26:56.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:26:56.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:26:56.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:26:56.894 INFO:tasks.workunit.client.0.trial150.stdout:2 ) state now: "active+clean" 2026-02-22T20:26:56.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 2 ') state now: ' '"active+clean"' 2026-02-22T20:26:56.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:26:56.894 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:26:56.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:26:56.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 2 == 4 ']' 2026-02-22T20:26:56.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:26:57.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:26:57.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:26:57.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:26:57.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:26:57.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:26:57.271 INFO:tasks.workunit.client.0.trial150.stdout:3 ) state now: "active+clean" 2026-02-22T20:26:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 3 ') state now: ' '"active+clean"' 2026-02-22T20:26:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:26:57.272 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:26:57.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:26:57.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 3 == 4 ']' 2026-02-22T20:26:57.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:26:57.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:26:57.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:26:57.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:26:57.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:26:57.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:26:57.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 4 ') state now: ' '"active+clean"' 2026-02-22T20:26:57.646 INFO:tasks.workunit.client.0.trial150.stdout:4 ) state now: "active+clean" 2026-02-22T20:26:57.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:26:57.646 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:26:57.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:26:57.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 4 == 4 ']' 2026-02-22T20:26:57.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:410: initiate_and_fetch_state: echo 'loop --------> ' 4 2026-02-22T20:26:57.647 INFO:tasks.workunit.client.0.trial150.stdout:loop --------> 4 2026-02-22T20:26:57.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:26:57.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:26:57.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:26:57.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:26:57.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:26:58.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:26:58.021 INFO:tasks.workunit.client.0.trial150.stdout:5 ) state now: "active+clean" 2026-02-22T20:26:58.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 5 ') state now: ' '"active+clean"' 2026-02-22T20:26:58.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:26:58.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:26:58.022 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:26:58.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 5 == 4 ']' 2026-02-22T20:26:58.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:26:58.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:26:58.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:26:58.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:26:58.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:26:58.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:26:58.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 6 ') state now: ' '"active+clean"' 2026-02-22T20:26:58.397 INFO:tasks.workunit.client.0.trial150.stdout:6 ) state now: "active+clean" 2026-02-22T20:26:58.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:26:58.398 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:26:58.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:26:58.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 6 == 4 ']' 2026-02-22T20:26:58.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:26:58.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:26:58.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:26:58.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:26:58.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:26:58.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:26:58.774 INFO:tasks.workunit.client.0.trial150.stdout:7 ) state now: "active+clean" 2026-02-22T20:26:58.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 7 ') state now: ' '"active+clean"' 2026-02-22T20:26:58.775 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:26:58.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:26:58.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:26:58.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 7 == 4 ']' 2026-02-22T20:26:58.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:26:59.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:26:59.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:26:59.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:26:59.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:26:59.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:26:59.147 INFO:tasks.workunit.client.0.trial150.stdout:8 ) state now: "active+clean" 2026-02-22T20:26:59.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 8 ') state now: ' '"active+clean"' 2026-02-22T20:26:59.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:26:59.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:26:59.148 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:26:59.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 8 == 4 ']' 2026-02-22T20:26:59.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:26:59.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:26:59.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:26:59.449 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:26:59.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:26:59.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:26:59.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 9 ') state now: ' '"active+clean"' 2026-02-22T20:26:59.523 INFO:tasks.workunit.client.0.trial150.stdout:9 ) state now: "active+clean" 2026-02-22T20:26:59.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:26:59.523 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:26:59.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:26:59.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 9 == 4 ']' 2026-02-22T20:26:59.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:26:59.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:26:59.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:26:59.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:26:59.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:26:59.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:26:59.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 10 ') state now: ' '"active+clean"' 2026-02-22T20:26:59.899 INFO:tasks.workunit.client.0.trial150.stdout:10 ) state now: "active+clean" 2026-02-22T20:26:59.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:26:59.900 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:26:59.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:26:59.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 0 == 4 ']' 2026-02-22T20:26:59.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:27:00.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:27:00.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:27:00.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:27:00.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:27:00.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:27:00.273 INFO:tasks.workunit.client.0.trial150.stdout:11 ) state now: "active+clean" 2026-02-22T20:27:00.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 11 ') state now: ' '"active+clean"' 2026-02-22T20:27:00.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:27:00.274 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:27:00.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:27:00.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 1 == 4 ']' 2026-02-22T20:27:00.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:27:00.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:27:00.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:27:00.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:27:00.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:27:00.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:27:00.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 12 ') state now: ' '"active+clean"' 2026-02-22T20:27:00.647 INFO:tasks.workunit.client.0.trial150.stdout:12 ) state now: "active+clean" 2026-02-22T20:27:00.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:27:00.648 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:27:00.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:27:00.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 2 == 4 ']' 2026-02-22T20:27:00.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:27:00.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:27:00.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:27:00.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:27:00.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:27:01.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:27:01.023 INFO:tasks.workunit.client.0.trial150.stdout:13 ) state now: "active+clean" 2026-02-22T20:27:01.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 13 ') state now: ' '"active+clean"' 2026-02-22T20:27:01.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:27:01.024 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:27:01.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:27:01.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 3 == 4 ']' 2026-02-22T20:27:01.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:27:01.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:27:01.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:27:01.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:27:01.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:27:01.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:27:01.398 INFO:tasks.workunit.client.0.trial150.stdout:14 ) state now: "active+clean" 2026-02-22T20:27:01.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 14 ') state now: ' '"active+clean"' 2026-02-22T20:27:01.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:27:01.399 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:27:01.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:27:01.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 4 == 4 ']' 2026-02-22T20:27:01.400 INFO:tasks.workunit.client.0.trial150.stdout:loop --------> 14 2026-02-22T20:27:01.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:410: initiate_and_fetch_state: echo 'loop --------> ' 14 2026-02-22T20:27:01.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:27:01.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:27:01.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:27:01.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:27:01.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:27:01.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:27:01.774 INFO:tasks.workunit.client.0.trial150.stdout:15 ) state now: "active+clean" 2026-02-22T20:27:01.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 15 ') state now: ' '"active+clean"' 2026-02-22T20:27:01.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:27:01.775 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:27:01.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:27:01.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 5 == 4 ']' 2026-02-22T20:27:01.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:27:02.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:27:02.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:27:02.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:27:02.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:27:02.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:27:02.147 INFO:tasks.workunit.client.0.trial150.stdout:16 ) state now: "active+clean" 2026-02-22T20:27:02.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 16 ') state now: ' '"active+clean"' 2026-02-22T20:27:02.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:27:02.147 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:27:02.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:27:02.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 6 == 4 ']' 2026-02-22T20:27:02.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:27:02.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:27:02.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:27:02.449 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:27:02.449 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:27:02.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean"' 2026-02-22T20:27:02.523 INFO:tasks.workunit.client.0.trial150.stdout:17 ) state now: "active+clean" 2026-02-22T20:27:02.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 17 ') state now: ' '"active+clean"' 2026-02-22T20:27:02.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:27:02.524 INFO:tasks.workunit.client.0.trial150.stdout:"active+clean" 2026-02-22T20:27:02.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:406: initiate_and_fetch_state: echo '"active+clean"' 2026-02-22T20:27:02.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:409: initiate_and_fetch_state: '[' 7 == 4 ']' 2026-02-22T20:27:02.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:412: initiate_and_fetch_state: sleep 0.3 2026-02-22T20:27:02.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i++ )) 2026-02-22T20:27:02.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:396: initiate_and_fetch_state: (( i < 80 )) 2026-02-22T20:27:02.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: ceph pg 1.0 query --format json 2026-02-22T20:27:02.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: jq .state 2026-02-22T20:27:02.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:398: initiate_and_fetch_state: st='"active+clean+scrubbing+deep"' 2026-02-22T20:27:02.899 INFO:tasks.workunit.client.0.trial150.stdout:18 ) state now: "active+clean+scrubbing+deep" 2026-02-22T20:27:02.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:399: initiate_and_fetch_state: echo 18 ') state now: ' '"active+clean+scrubbing+deep"' 2026-02-22T20:27:02.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:401: initiate_and_fetch_state: case "$st" in 2026-02-22T20:27:02.900 INFO:tasks.workunit.client.0.trial150.stdout:found scrub 2026-02-22T20:27:02.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:403: initiate_and_fetch_state: echo 'found scrub' 2026-02-22T20:27:02.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:403: initiate_and_fetch_state: return 0 2026-02-22T20:27:02.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:478: TEST_auto_repair_bluestore_tag: r=0 2026-02-22T20:27:02.901 INFO:tasks.workunit.client.0.trial150.stdout:initiate_and_fetch_state ret: 0 2026-02-22T20:27:02.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:479: TEST_auto_repair_bluestore_tag: echo 'initiate_and_fetch_state ret: ' 0 2026-02-22T20:27:02.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:480: TEST_auto_repair_bluestore_tag: set_config osd td/osd-scrub-repair osd_scrub_sleep 0 2026-02-22T20:27:02.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:27:02.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=td/osd-scrub-repair 2026-02-22T20:27:02.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=osd_scrub_sleep 2026-02-22T20:27:02.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=0 2026-02-22T20:27:02.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:27:02.903 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.td/osd-scrub-repair 2026-02-22T20:27:02.903 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.td/osd-scrub-repair 2026-02-22T20:27:02.903 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.td/osd-scrub-repair ']' 2026-02-22T20:27:02.903 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:27:02.904 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:02.904 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:02.904 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.td/osd-scrub-repair.asok 2026-02-22T20:27:02.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.td/osd-scrub-repair.asok config set osd_scrub_sleep 0 2026-02-22T20:27:02.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test == true 2026-02-22T20:27:02.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 1159: test: ==: unary operator expected 2026-02-22T20:27:02.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:481: TEST_auto_repair_bluestore_tag: '[' 0 -ne 0 ']' 2026-02-22T20:27:02.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:485: TEST_auto_repair_bluestore_tag: wait_end_of_scrub 1 1.0 2026-02-22T20:27:02.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:420: wait_end_of_scrub: local the_osd=osd.1 2026-02-22T20:27:02.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:421: wait_end_of_scrub: local pgid=1.0 2026-02-22T20:27:02.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i=0 )) 2026-02-22T20:27:02.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-22T20:27:02.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-22T20:27:02.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-22T20:27:03.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-22T20:27:03.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-22T20:27:03.019 INFO:tasks.workunit.client.0.trial150.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-22T20:27:03.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-22T20:27:03.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 0 == 4 ']' 2026-02-22T20:27:03.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-22T20:27:03.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-22T20:27:03.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-22T20:27:03.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-22T20:27:03.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-22T20:27:03.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-22T20:27:03.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-22T20:27:03.396 INFO:tasks.workunit.client.0.trial150.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-22T20:27:03.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-22T20:27:03.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 1 == 4 ']' 2026-02-22T20:27:03.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-22T20:27:03.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-22T20:27:03.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-22T20:27:03.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-22T20:27:03.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-22T20:27:03.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-22T20:27:03.770 INFO:tasks.workunit.client.0.trial150.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-22T20:27:03.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-22T20:27:03.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-22T20:27:03.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 2 == 4 ']' 2026-02-22T20:27:03.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-22T20:27:04.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-22T20:27:04.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-22T20:27:04.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-22T20:27:04.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-22T20:27:04.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-22T20:27:04.146 INFO:tasks.workunit.client.0.trial150.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-22T20:27:04.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-22T20:27:04.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-22T20:27:04.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 3 == 4 ']' 2026-02-22T20:27:04.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-22T20:27:04.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-22T20:27:04.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-22T20:27:04.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-22T20:27:04.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-22T20:27:04.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean+scrubbing+deep"' 2026-02-22T20:27:04.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean+scrubbing+deep"' 2026-02-22T20:27:04.524 INFO:tasks.workunit.client.0.trial150.stdout:wait-scrub-end state now: "active+clean+scrubbing+deep" 2026-02-22T20:27:04.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean+scrubbing+deep" =~ (.*scrubbing.*) ]] 2026-02-22T20:27:04.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:427: wait_end_of_scrub: '[' 4 == 4 ']' 2026-02-22T20:27:04.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:428: wait_end_of_scrub: flush_pg_stats 2026-02-22T20:27:04.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:27:04.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:27:04.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:27:04.688 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:27:04.688 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:27:04.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:27:04.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:04.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:27:04.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215111 2026-02-22T20:27:04.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215111 2026-02-22T20:27:04.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215111' 2026-02-22T20:27:04.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:04.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:27:04.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672970 2026-02-22T20:27:04.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672970 2026-02-22T20:27:04.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215111 1-42949672970' 2026-02-22T20:27:04.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:04.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:27:04.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509449 2026-02-22T20:27:04.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509449 2026-02-22T20:27:04.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215111 1-42949672970 2-64424509449' 2026-02-22T20:27:04.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:04.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215111 2026-02-22T20:27:04.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:04.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:27:04.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215111 2026-02-22T20:27:04.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:04.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215111 2026-02-22T20:27:04.917 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 103079215111 2026-02-22T20:27:04.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215111' 2026-02-22T20:27:04.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:05.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215110 -lt 103079215111 2026-02-22T20:27:05.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:27:06.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:27:06.078 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:06.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215110 -lt 103079215111 2026-02-22T20:27:06.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:27:07.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:27:07.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:07.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215111 -lt 103079215111 2026-02-22T20:27:07.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:07.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672970 2026-02-22T20:27:07.401 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:07.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:27:07.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672970 2026-02-22T20:27:07.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:07.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672970 2026-02-22T20:27:07.404 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672970 2026-02-22T20:27:07.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2026-02-22T20:27:07.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:27:07.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672970 -lt 42949672970 2026-02-22T20:27:07.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:07.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509449 2026-02-22T20:27:07.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:07.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:27:07.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509449 2026-02-22T20:27:07.578 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:07.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509449 2026-02-22T20:27:07.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2026-02-22T20:27:07.579 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509449 2026-02-22T20:27:07.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:27:07.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509449 -lt 64424509449 2026-02-22T20:27:07.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:430: wait_end_of_scrub: sleep 0.3 2026-02-22T20:27:08.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i++ )) 2026-02-22T20:27:08.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:423: wait_end_of_scrub: (( i < 40 )) 2026-02-22T20:27:08.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: ceph pg 1.0 query --format json 2026-02-22T20:27:08.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: jq .state 2026-02-22T20:27:08.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:424: wait_end_of_scrub: st='"active+clean"' 2026-02-22T20:27:08.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:425: wait_end_of_scrub: echo 'wait-scrub-end state now: ' '"active+clean"' 2026-02-22T20:27:08.120 INFO:tasks.workunit.client.0.trial150.stdout:wait-scrub-end state now: "active+clean" 2026-02-22T20:27:08.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2026-02-22T20:27:08.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:426: wait_end_of_scrub: break 2026-02-22T20:27:08.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:433: wait_end_of_scrub: [[ "active+clean" =~ (.*scrubbing.*) ]] 2026-02-22T20:27:08.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:438: wait_end_of_scrub: return 0 2026-02-22T20:27:08.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:486: TEST_auto_repair_bluestore_tag: ceph pg dump pgs 2026-02-22T20:27:08.267 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:27:08.268 INFO:tasks.workunit.client.0.trial150.stdout:1.0 1 0 0 0 0 7 0 0 1 0 1 active+clean+scrubbing+deep 2026-02-22T20:27:02.578821+0000 20'1 25:52 [1,0] 1 [1,0] 1 0'0 2026-02-22T20:26:41.821166+0000 0'0 2026-02-11T15:38:46.058440+0000 0 0 deep scrubbing for 3s 0 0 2026-02-22T20:27:08.268 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:27:08.268 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:27:08.269 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:27:08.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:490: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2026-02-22T20:27:08.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:27:08.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:27:08.278 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-22T20:27:08.278 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:27:08.279 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:27:08.279 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:27:08.279 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:27:08.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:27:08.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:27:08.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:27:08.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:490: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:27:08.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:27:08.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:27:08.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:27:08.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:27:08.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:27:08.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:27:08.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:27:08.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:27:08.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:27:08.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:27:08.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:27:08.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:27:08.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:27:08.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:27:08.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:27:08.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:27:08.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:27:08.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:27:08.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:27:08.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:27:08.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:27:08.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:27:08.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-22T20:27:08.980 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:27:08.981 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:27:09.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-22T20:27:09.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:27:09.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:27:09.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:27:09.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:27:09.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:27:09.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:27:09.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:27:09.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:27:09.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:27:09.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:27:09.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:27:09.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:27:09.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:27:09.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:27:09.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:27:09.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:27:09.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:27:09.066 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:27:09.066 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:09.066 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:09.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:09.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:09.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:27:09.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:27:09.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:27:09.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:27:09.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:27:09.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:27:09.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:27:09.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:27:09.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-22T20:27:09.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:27:09.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:27:09.068 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:27:09.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-22T20:27:09.069 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:27:09.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:27:09.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:27:09.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:27:09.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:27:09.085 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:09.078+0000 7fb0e14bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:09.086 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:09.082+0000 7fb0e14bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:09.087 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:09.082+0000 7fb0e14bea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:09.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:27:09.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:27:09.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:27:09.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:27:09.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:27:09.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:09.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:27:09.226 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:27:09.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:09.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:27:09.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:27:09.523 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:09.518+0000 7fb0e14bea40 -1 Falling back to public interface 2026-02-22T20:27:10.126 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:10.122+0000 7fb0e14bea40 -1 osd.0 25 log_to_monitors true 2026-02-22T20:27:10.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:27:10.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:10.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:27:10.388 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:27:10.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:10.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:27:10.546 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [24,25) [v2:127.0.0.1:6802/1150566506,v1:127.0.0.1:6803/1150566506] [v2:127.0.0.1:6804/1150566506,v1:127.0.0.1:6805/1150566506] exists,up 242d2df6-2d7b-4b4a-a2b3-88106e2b9404 2026-02-22T20:27:10.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:27:10.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:27:10.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:27:10.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:27:10.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:27:10.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:27:10.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:27:10.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:27:10.548 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:27:10.548 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:27:10.548 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:27:10.548 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:27:10.548 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:27:10.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:27:10.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:27:10.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:27:10.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:27:10.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:27:10.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:27:10.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:27:10.821 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:27:10.821 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:27:10.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:27:10.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:10.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:27:10.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-22T20:27:10.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-22T20:27:10.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586' 2026-02-22T20:27:10.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:10.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:27:10.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672972 2026-02-22T20:27:10.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672972 2026-02-22T20:27:10.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-42949672972' 2026-02-22T20:27:10.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:10.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:27:11.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509451 2026-02-22T20:27:11.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509451 2026-02-22T20:27:11.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-124554051586 1-42949672972 2-64424509451' 2026-02-22T20:27:11.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:11.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-124554051586 2026-02-22T20:27:11.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:11.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:27:11.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-124554051586 2026-02-22T20:27:11.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:11.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-22T20:27:11.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 124554051586' 2026-02-22T20:27:11.047 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 124554051586 2026-02-22T20:27:11.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:11.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 124554051586 2026-02-22T20:27:11.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:27:12.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:27:12.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:12.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 124554051586 2026-02-22T20:27:12.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:27:13.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:27:13.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:13.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051586 -lt 124554051586 2026-02-22T20:27:13.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:13.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672972 2026-02-22T20:27:13.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:13.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:27:13.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672972 2026-02-22T20:27:13.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:13.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672972 2026-02-22T20:27:13.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2026-02-22T20:27:13.534 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672972 2026-02-22T20:27:13.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:27:13.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672972 -lt 42949672972 2026-02-22T20:27:13.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:13.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509451 2026-02-22T20:27:13.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:13.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:27:13.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509451 2026-02-22T20:27:13.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:13.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509451 2026-02-22T20:27:13.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2026-02-22T20:27:13.703 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509451 2026-02-22T20:27:13.703 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:27:13.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509452 -lt 64424509451 2026-02-22T20:27:13.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:27:13.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:27:13.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:27:14.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:27:14.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:27:14.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:27:14.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:27:14.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:27:14.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:27:14.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:27:14.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:27:14.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:27:14.224 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:27:14.224 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:27:14.224 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:27:14.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:27:14.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:27:14.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:27:14.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:491: TEST_auto_repair_bluestore_tag: get_not_primary testpool SOMETHING 2026-02-22T20:27:14.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:27:14.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:27:14.415 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool SOMETHING 2026-02-22T20:27:14.415 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:27:14.415 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:27:14.416 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:27:14.416 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:27:14.574 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:27:14.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool SOMETHING 2026-02-22T20:27:14.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:27:14.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:491: TEST_auto_repair_bluestore_tag: objectstore_tool td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-22T20:27:14.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:27:14.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:27:14.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:27:14.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:27:14.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-22T20:27:14.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:27:14.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:27:14.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:27:14.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:27:14.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:27:14.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:27:14.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:27:14.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:27:14.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:27:14.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:27:15.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:27:15.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-22T20:27:15.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:27:15.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:27:15.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:27:15.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:27:15.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:27:15.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING get-bytes td/osd-scrub-repair/COPY 2026-02-22T20:27:15.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-22T20:27:15.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:27:15.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:27:15.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:27:15.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:27:15.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:27:15.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:27:15.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:27:15.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:27:15.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:27:15.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:27:15.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:27:15.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:27:15.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:27:15.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:27:15.213 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:27:15.213 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:27:15.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:27:15.214 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:27:15.214 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:15.214 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:15.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:15.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:15.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:27:15.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:27:15.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:27:15.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:27:15.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:27:15.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:27:15.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:27:15.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:27:15.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq' 2026-02-22T20:27:15.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:27:15.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:27:15.216 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:27:15.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --osd-op-queue=wpq 2026-02-22T20:27:15.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:27:15.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:27:15.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:27:15.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:27:15.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:27:15.231 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:15.226+0000 7fcc153b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:15.233 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:15.226+0000 7fcc153b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:15.233 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:15.230+0000 7fcc153b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:15.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:27:15.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:27:15.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:27:15.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:27:15.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:27:15.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:15.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:27:15.377 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:27:15.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:15.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:27:15.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:27:15.623 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:15.618+0000 7fcc153b5a40 -1 Falling back to public interface 2026-02-22T20:27:16.232 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:16.226+0000 7fcc153b5a40 -1 osd.0 30 log_to_monitors true 2026-02-22T20:27:16.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:27:16.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:16.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:27:16.538 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:27:16.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:16.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:27:16.690 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 34 up_thru 0 down_at 31 last_clean_interval [29,30) [v2:127.0.0.1:6802/2065934756,v1:127.0.0.1:6803/2065934756] [v2:127.0.0.1:6804/2065934756,v1:127.0.0.1:6805/2065934756] exists,up 242d2df6-2d7b-4b4a-a2b3-88106e2b9404 2026-02-22T20:27:16.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:27:16.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:27:16.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:27:16.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:27:16.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:27:16.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:27:16.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:27:16.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:27:16.691 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:27:16.692 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:27:16.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:27:16.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:27:16.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:27:16.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:27:16.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:27:16.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:27:16.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:27:16.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:27:16.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:27:16.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:27:16.960 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:27:16.960 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:27:16.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:27:16.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:16.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:27:17.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=146028888066 2026-02-22T20:27:17.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 146028888066 2026-02-22T20:27:17.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066' 2026-02-22T20:27:17.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:17.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:27:17.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672974 2026-02-22T20:27:17.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672974 2026-02-22T20:27:17.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066 1-42949672974' 2026-02-22T20:27:17.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:17.106 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:27:17.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509453 2026-02-22T20:27:17.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509453 2026-02-22T20:27:17.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-146028888066 1-42949672974 2-64424509453' 2026-02-22T20:27:17.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:17.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-146028888066 2026-02-22T20:27:17.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:17.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:27:17.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-146028888066 2026-02-22T20:27:17.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:17.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=146028888066 2026-02-22T20:27:17.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 146028888066' 2026-02-22T20:27:17.181 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 146028888066 2026-02-22T20:27:17.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:17.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 146028888066 2026-02-22T20:27:17.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:27:18.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:27:18.343 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:18.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 146028888066 2026-02-22T20:27:18.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:27:19.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:27:19.503 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:19.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 146028888066 -lt 146028888066 2026-02-22T20:27:19.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:19.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672974 2026-02-22T20:27:19.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:19.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:27:19.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672974 2026-02-22T20:27:19.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:19.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672974 2026-02-22T20:27:19.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2026-02-22T20:27:19.666 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672974 2026-02-22T20:27:19.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:27:19.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672974 -lt 42949672974 2026-02-22T20:27:19.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:19.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509453 2026-02-22T20:27:19.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:19.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:27:19.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509453 2026-02-22T20:27:19.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:19.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509453 2026-02-22T20:27:19.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2026-02-22T20:27:19.830 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509453 2026-02-22T20:27:19.830 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:27:19.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509454 -lt 64424509453 2026-02-22T20:27:19.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:27:19.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:27:19.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:27:20.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:27:20.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:27:20.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:27:20.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:27:20.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:27:20.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:27:20.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:27:20.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:27:20.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:27:20.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:27:20.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:27:20.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:27:20.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:27:20.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:27:20.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:27:20.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:492: TEST_auto_repair_bluestore_tag: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:27:20.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:493: TEST_auto_repair_bluestore_tag: grep scrub_finish td/osd-scrub-repair/osd.1.log 2026-02-22T20:27:20.533 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:27:05.578+0000 7f9c8aef3640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: scrub_finish before flags: AUTO_REPAIR. repair state: no-repair. deep_scrub_on_error: 0 2026-02-22T20:27:20.534 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:27:05.578+0000 7f9c8aef3640 10 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep(periodic-regular,auto-repair)] scrubber: _scrub_finish info stats: valid m_is_repair: 1 2026-02-22T20:27:20.534 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:27:05.578+0000 7f9c8aef3640 15 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish: 1 errors. 1 errors fixed 2026-02-22T20:27:20.534 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:27:05.578+0000 7f9c8aef3640 20 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish All may be fixed 2026-02-22T20:27:20.534 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:27:05.578+0000 7f9c8aef3640 19 osd.1 pg_epoch: 25 pg[1.0( v 20'1 (0'0,20'1] local-lis/les=24/25 n=1 ec=16/16 lis/c=24/24 les/c/f=25/25/0 sis=24) [1,0] r=0 lpr=24 crt=20'1 lcod 0'0 mlcod 0'0 active+scrubbing+deep+repair(periodic-regular,auto-repair) mbc={255={(1+0)=1}}] scrubber: scrub_finish shard 1 num_omap_bytes = 0 num_omap_keys = 0 2026-02-22T20:27:20.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:27:20.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:27:20.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:27:20.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:27:20.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:27:20.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:27:20.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:27:20.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:27:20.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:27:20.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:27:20.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:27:20.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:27:20.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:27:20.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:27:20.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:27:20.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:27:20.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:27:20.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:27:20.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:27:20.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:27:20.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:27:20.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:27:20.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:27:20.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:27:20.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:20.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:20.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:27:20.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:27:20.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:27:20.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:27:20.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:27:20.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:27:20.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:27:20.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:27:20.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:27:20.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:27:20.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:27:20.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:27:20.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:27:20.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:27:20.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:27:20.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:27:20.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:27:20.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:27:20.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:27:20.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:27:20.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:27:20.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:27:20.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:27:20.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:27:20.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:27:20.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:27:20.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:27:20.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:27:20.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:27:20.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:27:20.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:20.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:20.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:27:20.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:27:20.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:27:20.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:27:20.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:27:20.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:20.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:20.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:27:20.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:27:20.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:27:20.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:27:20.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:27:20.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_appends td/osd-scrub-repair 2026-02-22T20:27:20.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:369: TEST_auto_repair_erasure_coded_appends: auto_repair_erasure_coded td/osd-scrub-repair false 2026-02-22T20:27:20.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:328: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-22T20:27:20.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: auto_repair_erasure_coded: local allow_overwrites=false 2026-02-22T20:27:20.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: auto_repair_erasure_coded: local poolname=ecpool 2026-02-22T20:27:20.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-22T20:27:20.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:27:20.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:27:20.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:27:20.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:27:20.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:27:20.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:27:20.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:27:20.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:27:20.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:27:20.707 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:27:20.707 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:20.707 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:20.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:20.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:27:20.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:27:20.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:27:20.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:27:20.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:27:20.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:27:20.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:27:20.739 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:27:20.739 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:27:20.739 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:27:20.739 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:27:20.739 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:20.739 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:20.739 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:27:20.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:27:20.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:27:20.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:27:20.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:27:20.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:27:20.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:27:20.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:27:20.797 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:27:20.797 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:27:20.797 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:27:20.797 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:27:20.797 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:20.797 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:20.797 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:27:20.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:27:20.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:27:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:334: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-22T20:27:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:27:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:27:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:27:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:27:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:27:20.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:27:20.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:27:20.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:27:20.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:27:20.946 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:27:20.947 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:20.947 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:20.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:20.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:27:20.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:27:20.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:27:20.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: seq 0 2 2026-02-22T20:27:20.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:27:20.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:27:20.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:27:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:27:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:27:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:27:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:27:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:27:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:27:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:27:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:27:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:27:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:27:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:27:20.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:27:20.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:27:20.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:27:20.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:27:20.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:27:20.972 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:27:20.972 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:20.972 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:20.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:20.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:20.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:27:20.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:27:20.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:27:20.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:27:20.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:27:20.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:27:20.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:27:20.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:27:20.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:27:20.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:27:20.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:27:20.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:27:20.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:27:20.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3be3b032-2b10-423b-acbe-0de82dc1f4c4 2026-02-22T20:27:20.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 3be3b032-2b10-423b-acbe-0de82dc1f4c4' 2026-02-22T20:27:20.975 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 3be3b032-2b10-423b-acbe-0de82dc1f4c4 2026-02-22T20:27:20.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:27:20.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCoZptpCaNtOhAAWMNPWz4E4LVHYtEuC3FkdQ== 2026-02-22T20:27:20.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCoZptpCaNtOhAAWMNPWz4E4LVHYtEuC3FkdQ=="}' 2026-02-22T20:27:20.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3be3b032-2b10-423b-acbe-0de82dc1f4c4 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:27:21.073 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:27:21.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:27:21.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCoZptpCaNtOhAAWMNPWz4E4LVHYtEuC3FkdQ== --osd-uuid 3be3b032-2b10-423b-acbe-0de82dc1f4c4 2026-02-22T20:27:21.100 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:21.094+0000 7f33504b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:21.102 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:21.098+0000 7f33504b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:21.103 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:21.098+0000 7f33504b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:21.103 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:21.098+0000 7f33504b4a40 -1 bdev(0x55d4ee746c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:27:21.103 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:21.098+0000 7f33504b4a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:27:21.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:27:21.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:27:21.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:27:21.837 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:27:21.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:27:22.039 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:27:22.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:27:22.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:27:22.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:27:22.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:27:22.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:27:22.057 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:22.050+0000 7fbbe4cdda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:22.058 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:22.054+0000 7fbbe4cdda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:22.059 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:22.054+0000 7fbbe4cdda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:22.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:27:22.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:27:22.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:27:22.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:27:22.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:27:22.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:22.201 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:27:22.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:27:22.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:22.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:27:22.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:27:22.423 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:22.418+0000 7fbbe4cdda40 -1 Falling back to public interface 2026-02-22T20:27:22.980 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:22.974+0000 7fbbe4cdda40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:27:23.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:27:23.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:23.360 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:27:23.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:27:23.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:23.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:27:23.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:27:24.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:27:24.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:24.523 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:27:24.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:27:24.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:24.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:27:24.679 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/112296850,v1:127.0.0.1:6803/112296850] [v2:127.0.0.1:6804/112296850,v1:127.0.0.1:6805/112296850] exists,up 3be3b032-2b10-423b-acbe-0de82dc1f4c4 2026-02-22T20:27:24.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:27:24.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:27:24.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:27:24.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:27:24.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:27:24.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:27:24.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:27:24.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:27:24.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:27:24.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:27:24.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:27:24.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:27:24.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:27:24.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:27:24.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:27:24.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:27:24.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:27:24.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:27:24.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:27:24.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:27:24.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:27:24.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:27:24.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:27:24.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:27:24.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:27:24.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:27:24.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:27:24.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:27:24.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e7384c83-6b4f-49e0-a22e-55fb99e7b238 2026-02-22T20:27:24.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e7384c83-6b4f-49e0-a22e-55fb99e7b238' 2026-02-22T20:27:24.687 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 e7384c83-6b4f-49e0-a22e-55fb99e7b238 2026-02-22T20:27:24.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:27:24.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCsZptp9k9gKRAA2Qf8YWcIIIeUEgI0BEwkhg== 2026-02-22T20:27:24.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCsZptp9k9gKRAA2Qf8YWcIIIeUEgI0BEwkhg=="}' 2026-02-22T20:27:24.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e7384c83-6b4f-49e0-a22e-55fb99e7b238 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:27:24.857 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:27:24.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:27:24.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCsZptp9k9gKRAA2Qf8YWcIIIeUEgI0BEwkhg== --osd-uuid e7384c83-6b4f-49e0-a22e-55fb99e7b238 2026-02-22T20:27:24.885 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:24.878+0000 7fdf183dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:24.886 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:24.882+0000 7fdf183dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:24.887 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:24.882+0000 7fdf183dea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:24.888 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:24.882+0000 7fdf183dea40 -1 bdev(0x562320b1ec00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:27:24.888 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:24.882+0000 7fdf183dea40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:27:25.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:27:25.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:27:25.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:27:25.526 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:27:25.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:27:25.726 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:27:25.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:27:25.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:27:25.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:27:25.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:27:25.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:27:25.752 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:25.746+0000 7f0f32eeaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:25.754 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:25.750+0000 7f0f32eeaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:25.755 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:25.750+0000 7f0f32eeaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:25.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:27:25.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:27:25.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:27:25.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:27:25.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:27:25.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:25.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:27:25.888 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:27:25.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:25.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:27:26.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:27:26.167 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:26.162+0000 7f0f32eeaa40 -1 Falling back to public interface 2026-02-22T20:27:26.781 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:26.774+0000 7f0f32eeaa40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:27:27.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:27:27.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:27.044 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:27:27.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:27:27.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:27.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:27:27.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:27:28.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:27:28.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:28.209 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:27:28.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:27:28.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:28.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:27:28.363 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3500398689,v1:127.0.0.1:6811/3500398689] [v2:127.0.0.1:6812/3500398689,v1:127.0.0.1:6813/3500398689] exists,up e7384c83-6b4f-49e0-a22e-55fb99e7b238 2026-02-22T20:27:28.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:27:28.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:27:28.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:27:28.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:27:28.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:27:28.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:27:28.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:27:28.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:27:28.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:27:28.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:27:28.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:28.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:27:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:27:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:27:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:27:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:27:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:27:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:27:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:27:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:27:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:27:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:27:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:27:28.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:27:28.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d5103846-8b3d-42bd-86f5-b9bcaee68960 2026-02-22T20:27:28.370 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 d5103846-8b3d-42bd-86f5-b9bcaee68960 2026-02-22T20:27:28.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 d5103846-8b3d-42bd-86f5-b9bcaee68960' 2026-02-22T20:27:28.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:27:28.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCwZptpIOqEFhAAS+OsmFKM4zpBQOHIsNwJbA== 2026-02-22T20:27:28.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCwZptpIOqEFhAAS+OsmFKM4zpBQOHIsNwJbA=="}' 2026-02-22T20:27:28.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d5103846-8b3d-42bd-86f5-b9bcaee68960 -i td/osd-scrub-repair/2/new.json 2026-02-22T20:27:28.529 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:27:28.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:27:28.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCwZptpIOqEFhAAS+OsmFKM4zpBQOHIsNwJbA== --osd-uuid d5103846-8b3d-42bd-86f5-b9bcaee68960 2026-02-22T20:27:28.566 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:28.562+0000 7f91d817fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:28.567 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:28.562+0000 7f91d817fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:28.568 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:28.562+0000 7f91d817fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:28.569 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:28.562+0000 7f91d817fa40 -1 bdev(0x564f62dbac00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:27:28.569 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:28.562+0000 7f91d817fa40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:27:29.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:27:29.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:27:29.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:27:29.179 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:27:29.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:27:29.384 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:27:29.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:27:29.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:27:29.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:27:29.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:27:29.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:27:29.402 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:29.398+0000 7f49a531ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:29.403 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:29.398+0000 7f49a531ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:29.404 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:29.398+0000 7f49a531ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:29.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:27:29.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:27:29.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:27:29.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:27:29.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:27:29.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:29.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:27:29.546 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:27:29.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:29.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:27:29.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:27:29.800 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:29.794+0000 7f49a531ba40 -1 Falling back to public interface 2026-02-22T20:27:30.456 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:30.450+0000 7f49a531ba40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:27:30.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:27:30.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:30.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:27:30.707 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:27:30.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:30.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:27:30.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:27:31.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:27:31.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:31.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:27:31.870 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:27:31.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:31.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:27:32.028 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1515042962,v1:127.0.0.1:6819/1515042962] [v2:127.0.0.1:6820/1515042962,v1:127.0.0.1:6821/1515042962] exists,up d5103846-8b3d-42bd-86f5-b9bcaee68960 2026-02-22T20:27:32.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:27:32.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:27:32.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:27:32.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: auto_repair_erasure_coded: create_rbd_pool 2026-02-22T20:27:32.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:27:32.175 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:27:32.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:27:32.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:27:32.400 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:27:32.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:27:33.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:27:33.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:344: auto_repair_erasure_coded: wait_for_clean 2026-02-22T20:27:33.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:27:33.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:27:33.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:27:33.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:27:33.695 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:27:33.695 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:27:33.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:27:33.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:27:33.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:27:33.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:27:33.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:27:33.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:27:33.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:27:33.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:27:33.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:27:33.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:27:33.970 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:27:33.970 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:27:33.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:27:33.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:33.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:27:34.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:27:34.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:27:34.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:27:34.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:34.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:27:34.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-22T20:27:34.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-22T20:27:34.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-22T20:27:34.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:34.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:27:34.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574850 2026-02-22T20:27:34.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574850 2026-02-22T20:27:34.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667 2-55834574850' 2026-02-22T20:27:34.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:34.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:27:34.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:34.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:27:34.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:27:34.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:34.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:27:34.194 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:27:34.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:27:34.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:34.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:27:34.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:27:35.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:27:35.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:35.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-22T20:27:35.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:35.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-22T20:27:35.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:35.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:27:35.518 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-22T20:27:35.518 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:35.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-22T20:27:35.520 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705667 2026-02-22T20:27:35.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-22T20:27:35.520 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:27:35.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-22T20:27:35.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:35.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574850 2026-02-22T20:27:35.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:35.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:27:35.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574850 2026-02-22T20:27:35.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:35.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574850 2026-02-22T20:27:35.693 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574850 2026-02-22T20:27:35.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2026-02-22T20:27:35.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:27:35.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574850 -lt 55834574850 2026-02-22T20:27:35.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:27:35.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:27:35.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:27:36.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:27:36.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:27:36.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:27:36.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:27:36.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:27:36.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:27:36.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:27:36.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:27:36.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:27:36.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:27:36.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:27:36.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:27:36.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:27:36.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:27:36.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:27:36.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:347: auto_repair_erasure_coded: create_ec_pool ecpool false k=2 m=1 2026-02-22T20:27:36.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-22T20:27:36.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:27:36.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-22T20:27:36.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:27:36.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-22T20:27:36.600 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:27:36.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-22T20:27:36.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-22T20:27:36.905 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' created 2026-02-22T20:27:36.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:27:37.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-22T20:27:37.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:27:37.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:27:37.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:27:37.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:27:37.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:27:37.920 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:27:37.920 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:27:37.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:27:37.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:27:37.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:27:38.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:27:38.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:27:38.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:27:38.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:27:38.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:27:38.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:27:38.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:27:38.186 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:27:38.186 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:27:38.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:27:38.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:38.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:27:38.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-22T20:27:38.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-22T20:27:38.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-22T20:27:38.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:38.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:27:38.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-22T20:27:38.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-22T20:27:38.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669' 2026-02-22T20:27:38.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:38.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:27:38.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574852 2026-02-22T20:27:38.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574852 2026-02-22T20:27:38.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669 2-55834574852' 2026-02-22T20:27:38.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:38.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-22T20:27:38.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:38.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:27:38.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-22T20:27:38.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:38.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-22T20:27:38.417 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836485 2026-02-22T20:27:38.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-22T20:27:38.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:38.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-22T20:27:38.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:27:39.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:27:39.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:39.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-22T20:27:39.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:39.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-22T20:27:39.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:39.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:27:39.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-22T20:27:39.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:39.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-22T20:27:39.747 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705669 2026-02-22T20:27:39.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-22T20:27:39.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:27:39.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-22T20:27:39.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:39.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574852 2026-02-22T20:27:39.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:39.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:27:39.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574852 2026-02-22T20:27:39.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:39.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574852 2026-02-22T20:27:39.907 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574852 2026-02-22T20:27:39.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574852' 2026-02-22T20:27:39.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:27:40.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-22T20:27:40.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:27:40.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:27:40.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:27:40.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:27:40.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:27:40.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:27:40.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:27:40.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:27:40.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:27:40.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:27:40.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:27:40.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:27:40.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:27:40.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:27:40.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:27:40.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:27:40.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:27:40.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:27:40.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:27:40.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: auto_repair_erasure_coded: local payload=ABCDEF 2026-02-22T20:27:40.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:351: auto_repair_erasure_coded: echo ABCDEF 2026-02-22T20:27:40.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:27:40.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2026-02-22T20:27:40.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2026-02-22T20:27:40.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:27:40.619 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2026-02-22T20:27:40.619 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-22T20:27:40.619 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:27:40.619 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:27:40.619 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:27:40.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2026-02-22T20:27:40.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:27:40.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:27:40.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:27:41.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:27:41.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:27:41.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:27:41.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:27:41.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:27:41.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:27:41.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:27:41.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-22T20:27:41.428 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-22T20:27:41.428 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-22T20:27:41.428 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#-3:00000000:::deep_scrub_1.0:head#, (61) No data available 2026-02-22T20:27:41.428 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2026-02-22T20:27:41.429 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:27:41.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:27:41.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:27:41.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:27:41.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:27:41.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:27:41.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:27:41.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:27:41.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:27:41.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:27:41.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:27:41.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:27:41.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:27:41.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:27:41.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:27:41.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:27:41.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:27:41.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:27:41.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:27:41.539 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:27:41.539 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:41.539 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:41.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:41.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:41.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:27:41.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:27:41.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:27:41.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:27:41.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:27:41.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:27:41.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:27:41.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:27:41.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:27:41.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:27:41.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:27:41.541 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:27:41.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:27:41.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:27:41.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:27:41.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:27:41.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:27:41.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:27:41.557 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:41.549+0000 7fa5d7bb7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:41.558 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:41.553+0000 7fa5d7bb7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:41.559 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:41.553+0000 7fa5d7bb7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:41.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:27:41.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:27:41.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:27:41.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:27:41.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:27:41.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:41.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:27:41.695 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:27:41.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:41.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:27:41.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:27:41.972 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:41.965+0000 7fa5d7bb7a40 -1 Falling back to public interface 2026-02-22T20:27:42.614 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:42.609+0000 7fa5d7bb7a40 -1 osd.1 25 log_to_monitors true 2026-02-22T20:27:42.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:27:42.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:42.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:27:42.854 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:27:42.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:42.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:27:43.011 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 29 up_thru 29 down_at 26 last_clean_interval [9,25) [v2:127.0.0.1:6810/645639917,v1:127.0.0.1:6811/645639917] [v2:127.0.0.1:6812/645639917,v1:127.0.0.1:6813/645639917] exists,up e7384c83-6b4f-49e0-a22e-55fb99e7b238 2026-02-22T20:27:43.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:27:43.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:27:43.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:27:43.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:27:43.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:27:43.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:27:43.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:27:43.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:27:43.012 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:27:43.013 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:27:43.013 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:27:43.013 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:27:43.013 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:27:43.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:27:43.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:27:43.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:27:43.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:27:43.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:27:43.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:27:43.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:27:43.277 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:27:43.277 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:27:43.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:27:43.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:43.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:27:43.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-22T20:27:43.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-22T20:27:43.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-22T20:27:43.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:43.349 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:27:43.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051586 2026-02-22T20:27:43.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051586 2026-02-22T20:27:43.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-124554051586' 2026-02-22T20:27:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:43.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:27:43.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574854 2026-02-22T20:27:43.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574854 2026-02-22T20:27:43.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-124554051586 2-55834574854' 2026-02-22T20:27:43.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:43.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-22T20:27:43.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:43.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:27:43.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-22T20:27:43.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:43.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-22T20:27:43.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-22T20:27:43.497 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836487 2026-02-22T20:27:43.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:43.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-22T20:27:43.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:43.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-124554051586 2026-02-22T20:27:43.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:43.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:27:43.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-124554051586 2026-02-22T20:27:43.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:43.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051586 2026-02-22T20:27:43.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 124554051586' 2026-02-22T20:27:43.669 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 124554051586 2026-02-22T20:27:43.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:27:43.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051586 -lt 124554051586 2026-02-22T20:27:43.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:43.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574854 2026-02-22T20:27:43.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:43.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:27:43.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574854 2026-02-22T20:27:43.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:43.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574854 2026-02-22T20:27:43.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574854' 2026-02-22T20:27:43.841 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 55834574854 2026-02-22T20:27:43.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:27:44.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574853 -lt 55834574854 2026-02-22T20:27:44.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:27:45.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:27:45.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:27:45.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574853 -lt 55834574854 2026-02-22T20:27:45.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:27:46.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:27:46.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:27:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574854 -lt 55834574854 2026-02-22T20:27:46.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:27:46.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:27:46.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:27:46.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:27:46.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:27:46.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:27:46.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:27:46.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:27:46.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:27:46.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:27:46.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:27:46.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:27:46.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:27:46.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:27:46.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:27:46.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:27:46.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:27:46.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:27:46.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2026-02-22T20:27:46.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:27:46.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:27:46.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:27:46.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:27:47.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: auto_repair_erasure_coded: local pgid=2.0 2026-02-22T20:27:47.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:359: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2026-02-22T20:27:47.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:27:47.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:27:47.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:27:47.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:27:47.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:359: auto_repair_erasure_coded: wait_for_scrub 2.0 2026-02-22T20:27:36.902636+0000 2026-02-22T20:27:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:27:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:27:36.902636+0000 2026-02-22T20:27:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:27:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:27:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:27:47.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:27:47.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:27:47.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:27:47.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:27:47.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:27:47.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:27:36.902636+0000 '>' 2026-02-22T20:27:36.902636+0000 2026-02-22T20:27:47.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:27:48.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:27:48.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:27:48.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:27:48.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:27:48.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:27:48.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:27:48.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:27:48.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:27:36.902636+0000 '>' 2026-02-22T20:27:36.902636+0000 2026-02-22T20:27:48.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:27:49.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:27:49.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:27:49.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:27:49.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:27:49.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:27:49.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:27:49.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:27:49.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:27:36.902636+0000 '>' 2026-02-22T20:27:36.902636+0000 2026-02-22T20:27:49.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:27:50.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:27:50.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:27:50.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:27:50.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:27:50.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:27:50.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:27:50.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:27:50.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:27:36.902636+0000 '>' 2026-02-22T20:27:36.902636+0000 2026-02-22T20:27:50.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:27:51.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:27:51.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:27:51.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:27:51.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:27:51.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:27:51.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:27:51.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:27:52.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:27:36.902636+0000 '>' 2026-02-22T20:27:36.902636+0000 2026-02-22T20:27:52.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:27:53.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:27:53.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:27:53.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:27:53.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:27:53.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:27:53.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:27:53.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:27:53.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:27:36.902636+0000 '>' 2026-02-22T20:27:36.902636+0000 2026-02-22T20:27:53.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:27:54.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:27:54.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:27:54.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:27:54.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:27:54.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:27:54.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:27:54.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:27:54.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:27:50.662257+0000 '>' 2026-02-22T20:27:36.902636+0000 2026-02-22T20:27:54.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:27:54.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: auto_repair_erasure_coded: wait_for_clean 2026-02-22T20:27:54.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:27:54.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:27:54.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:27:54.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:27:54.333 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:27:54.333 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:27:54.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:27:54.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:27:54.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:27:54.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:27:54.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:27:54.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:27:54.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:27:54.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:27:54.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:27:54.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:27:54.598 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:27:54.598 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:27:54.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:27:54.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:54.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:27:54.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-22T20:27:54.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-22T20:27:54.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T20:27:54.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:54.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:27:54.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=124554051589 2026-02-22T20:27:54.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 124554051589 2026-02-22T20:27:54.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-124554051589' 2026-02-22T20:27:54.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:54.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:27:54.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574857 2026-02-22T20:27:54.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574857 2026-02-22T20:27:54.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-124554051589 2-55834574857' 2026-02-22T20:27:54.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:54.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-22T20:27:54.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:54.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:27:54.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-22T20:27:54.816 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:54.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-22T20:27:54.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T20:27:54.817 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836491 2026-02-22T20:27:54.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:54.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-22T20:27:54.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:27:55.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:27:55.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:56.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-22T20:27:56.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:56.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-124554051589 2026-02-22T20:27:56.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:56.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:27:56.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-124554051589 2026-02-22T20:27:56.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:56.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=124554051589 2026-02-22T20:27:56.148 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 124554051589 2026-02-22T20:27:56.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 124554051589' 2026-02-22T20:27:56.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:27:56.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 124554051589 -lt 124554051589 2026-02-22T20:27:56.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:56.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574857 2026-02-22T20:27:56.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:56.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:27:56.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574857 2026-02-22T20:27:56.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:56.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574857 2026-02-22T20:27:56.320 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574857 2026-02-22T20:27:56.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574857' 2026-02-22T20:27:56.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:27:56.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574857 -lt 55834574857 2026-02-22T20:27:56.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:27:56.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:27:56.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:27:56.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:27:56.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:27:56.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:27:56.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:27:56.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:27:56.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:27:56.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:27:56.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:27:56.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:27:56.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:27:56.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:27:56.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:27:57.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:27:57.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:27:57.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:27:57.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2026-02-22T20:27:57.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2026-02-22T20:27:57.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:27:57.020 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2026-02-22T20:27:57.020 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-22T20:27:57.021 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:27:57.021 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:27:57.021 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:27:57.180 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2026-02-22T20:27:57.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:27:57.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:27:57.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:27:57.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:27:57.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:27:57.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:27:57.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:27:57.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:27:57.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:27:57.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:27:57.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-22T20:27:57.725 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-22T20:27:57.726 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-22T20:27:57.726 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#-3:00000000:::deep_scrub_1.0:head#, (61) No data available 2026-02-22T20:27:57.726 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2026-02-22T20:27:57.726 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:27:57.726 INFO:tasks.workunit.client.0.trial150.stdout:hinfo_key 2026-02-22T20:27:57.727 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:27:57.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:27:57.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:27:57.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:27:57.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:27:57.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:27:57.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:27:57.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:27:57.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:27:57.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:27:57.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:27:57.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:27:57.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:27:57.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:27:57.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:27:57.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:27:57.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:27:57.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:27:57.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:27:57.834 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:27:57.834 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:27:57.834 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:27:57.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:57.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:27:57.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:27:57.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:27:57.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:27:57.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:27:57.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:27:57.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:27:57.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:27:57.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:27:57.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:27:57.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:27:57.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:27:57.835 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:27:57.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:27:57.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:27:57.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:27:57.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:27:57.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:27:57.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:27:57.853 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:57.845+0000 7f60b99dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:57.854 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:57.845+0000 7f60b99dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:57.855 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:57.849+0000 7f60b99dba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:27:57.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:27:57.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:27:57.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:27:57.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:27:57.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:27:57.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:57.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:27:57.994 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:27:57.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:57.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:27:58.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:27:58.256 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:58.249+0000 7f60b99dba40 -1 Falling back to public interface 2026-02-22T20:27:58.903 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:27:58.897+0000 7f60b99dba40 -1 osd.1 31 log_to_monitors true 2026-02-22T20:27:59.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:27:59.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:27:59.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:27:59.153 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:27:59.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:27:59.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:27:59.313 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [29,31) [v2:127.0.0.1:6810/2514467382,v1:127.0.0.1:6811/2514467382] [v2:127.0.0.1:6812/2514467382,v1:127.0.0.1:6813/2514467382] exists,up e7384c83-6b4f-49e0-a22e-55fb99e7b238 2026-02-22T20:27:59.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:27:59.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:27:59.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:27:59.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:27:59.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:27:59.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:27:59.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:27:59.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:27:59.314 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:27:59.314 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:27:59.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:27:59.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:27:59.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:27:59.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:27:59.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:27:59.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:27:59.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:27:59.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:27:59.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:27:59.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:27:59.588 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:27:59.588 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:27:59.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:27:59.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:59.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:27:59.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-22T20:27:59.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-22T20:27:59.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-22T20:27:59.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:59.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:27:59.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855362 2026-02-22T20:27:59.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855362 2026-02-22T20:27:59.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-150323855362' 2026-02-22T20:27:59.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:27:59.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:27:59.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574859 2026-02-22T20:27:59.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574859 2026-02-22T20:27:59.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-150323855362 2-55834574859' 2026-02-22T20:27:59.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:27:59.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-22T20:27:59.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:27:59.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:27:59.816 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-22T20:27:59.816 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:27:59.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-22T20:27:59.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-22T20:27:59.818 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836493 2026-02-22T20:27:59.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:27:59.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-22T20:27:59.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:28:00.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:28:00.986 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:01.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-22T20:28:01.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:28:02.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:28:02.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:02.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-22T20:28:02.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:02.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-150323855362 2026-02-22T20:28:02.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:02.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:28:02.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-150323855362 2026-02-22T20:28:02.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:02.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855362 2026-02-22T20:28:02.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2026-02-22T20:28:02.310 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 150323855362 2026-02-22T20:28:02.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:28:02.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855362 -lt 150323855362 2026-02-22T20:28:02.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:02.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574859 2026-02-22T20:28:02.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:02.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:28:02.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574859 2026-02-22T20:28:02.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:02.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574859 2026-02-22T20:28:02.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574859' 2026-02-22T20:28:02.471 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 55834574859 2026-02-22T20:28:02.472 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:28:02.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574859 -lt 55834574859 2026-02-22T20:28:02.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:28:02.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:28:02.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:28:02.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:28:02.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:28:02.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:28:02.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:28:02.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:28:02.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:28:02.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:28:02.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:28:02.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:28:02.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:28:02.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:28:02.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:28:03.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:28:03.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:28:03.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:28:03.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:28:03.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:28:03.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:28:03.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:28:03.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:28:03.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:28:03.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:28:03.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:28:03.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:28:03.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:28:03.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:28:03.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:28:03.287 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:28:03.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:28:03.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:28:03.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:28:03.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:28:03.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:28:03.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:28:03.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:28:03.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:28:03.292 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:28:03.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:28:03.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:28:03.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:28:03.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:28:03.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:03.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:03.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:28:03.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:28:03.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:28:03.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:28:03.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:28:03.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:28:03.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:28:03.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:28:03.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:28:03.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:28:03.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:28:03.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:28:03.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:28:03.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:28:03.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:28:03.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:28:03.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:28:03.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:28:03.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:28:03.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:28:03.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:28:03.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:28:03.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:28:03.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:28:03.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:28:03.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:28:03.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:28:03.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:28:03.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:28:03.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:28:03.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:03.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:03.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:28:03.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:28:03.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:28:03.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:28:03.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:28:03.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:03.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:03.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:28:03.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:28:03.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:28:03.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:28:03.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:28:03.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_auto_repair_erasure_coded_overwrites td/osd-scrub-repair 2026-02-22T20:28:03.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:373: TEST_auto_repair_erasure_coded_overwrites: '[' true = true ']' 2026-02-22T20:28:03.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:374: TEST_auto_repair_erasure_coded_overwrites: auto_repair_erasure_coded td/osd-scrub-repair true 2026-02-22T20:28:03.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:328: auto_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-22T20:28:03.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:329: auto_repair_erasure_coded: local allow_overwrites=true 2026-02-22T20:28:03.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:330: auto_repair_erasure_coded: local poolname=ecpool 2026-02-22T20:28:03.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:333: auto_repair_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-22T20:28:03.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:28:03.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:28:03.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:28:03.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:28:03.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:28:03.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:28:03.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:28:03.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:03.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:03.347 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:03.347 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:03.348 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:03.348 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:03.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:28:03.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:28:03.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:28:03.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:28:03.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:28:03.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:28:03.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:28:03.388 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:28:03.388 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:28:03.388 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:28:03.388 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:28:03.388 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:03.388 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:03.388 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:28:03.389 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:28:03.389 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:28:03.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:28:03.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:28:03.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:28:03.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:28:03.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:28:03.450 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:28:03.450 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:28:03.450 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:28:03.450 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:28:03.450 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:03.450 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:03.450 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:28:03.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:28:03.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:28:03.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:334: auto_repair_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-22T20:28:03.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:28:03.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:28:03.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:28:03.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:28:03.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:28:03.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:28:03.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:28:03.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:03.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:03.601 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:03.601 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:03.602 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:03.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:03.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:28:03.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:28:03.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:339: auto_repair_erasure_coded: local 'ceph_osd_args=--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:28:03.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: seq 0 2 2026-02-22T20:28:03.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:28:03.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 0 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:28:03.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:28:03.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:28:03.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:28:03.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:28:03.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:28:03.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:28:03.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:28:03.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:28:03.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:28:03.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:28:03.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:28:03.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:28:03.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:28:03.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:28:03.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:28:03.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:03.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:03.627 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:03.627 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:03.627 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:03.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:03.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:03.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:28:03.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:28:03.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:28:03.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:28:03.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:28:03.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:28:03.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:28:03.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:28:03.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:28:03.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:28:03.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:28:03.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:28:03.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:28:03.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a0c81fb1-1e0b-4ee8-9288-a82e38be5966 2026-02-22T20:28:03.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a0c81fb1-1e0b-4ee8-9288-a82e38be5966' 2026-02-22T20:28:03.630 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 a0c81fb1-1e0b-4ee8-9288-a82e38be5966 2026-02-22T20:28:03.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:28:03.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDTZptpZnDvJRAASaijPXskl8ClCgP60CnYNw== 2026-02-22T20:28:03.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDTZptpZnDvJRAASaijPXskl8ClCgP60CnYNw=="}' 2026-02-22T20:28:03.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a0c81fb1-1e0b-4ee8-9288-a82e38be5966 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:28:03.728 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:28:03.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:28:03.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDTZptpZnDvJRAASaijPXskl8ClCgP60CnYNw== --osd-uuid a0c81fb1-1e0b-4ee8-9288-a82e38be5966 2026-02-22T20:28:03.754 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:03.745+0000 7f1b0cf7fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:03.756 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:03.749+0000 7f1b0cf7fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:03.757 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:03.749+0000 7f1b0cf7fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:03.757 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:03.749+0000 7f1b0cf7fa40 -1 bdev(0x55cc22b7cc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:28:03.757 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:03.749+0000 7f1b0cf7fa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:28:04.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:28:04.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:28:04.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:28:04.498 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:28:04.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:28:04.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:28:04.704 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:28:04.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:28:04.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:28:04.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:28:04.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:28:04.731 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:04.725+0000 7f608a3e8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:04.732 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:04.725+0000 7f608a3e8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:04.734 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:04.725+0000 7f608a3e8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:04.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:28:04.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:28:04.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:28:04.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:28:04.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:28:04.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:04.865 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:28:04.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:28:04.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:04.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:28:05.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:05.124 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:05.117+0000 7f608a3e8a40 -1 Falling back to public interface 2026-02-22T20:28:05.692 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:05.685+0000 7f608a3e8a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:28:06.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:06.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:06.024 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:28:06.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:28:06.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:06.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:28:06.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:07.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:07.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:07.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:28:07.184 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:28:07.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:07.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:28:07.342 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2567205258,v1:127.0.0.1:6803/2567205258] [v2:127.0.0.1:6804/2567205258,v1:127.0.0.1:6805/2567205258] exists,up a0c81fb1-1e0b-4ee8-9288-a82e38be5966 2026-02-22T20:28:07.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:28:07.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:28:07.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:28:07.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:28:07.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:28:07.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:28:07.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:28:07.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:28:07.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:28:07.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:28:07.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:28:07.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:28:07.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:28:07.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:28:07.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:28:07.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:28:07.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:28:07.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:28:07.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:28:07.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:28:07.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:07.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:07.347 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:07.348 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:07.348 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:07.348 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:07.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:07.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:28:07.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:28:07.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:28:07.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:28:07.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:28:07.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:28:07.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:28:07.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:28:07.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:28:07.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:28:07.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:28:07.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:28:07.349 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:28:07.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6d9d5f15-2e39-4235-ba79-29e104d4f4a3 2026-02-22T20:28:07.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6d9d5f15-2e39-4235-ba79-29e104d4f4a3' 2026-02-22T20:28:07.350 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 6d9d5f15-2e39-4235-ba79-29e104d4f4a3 2026-02-22T20:28:07.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:28:07.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDXZptpbJMfFRAAI6rjwTCaXn0CS8vrcJQNew== 2026-02-22T20:28:07.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDXZptpbJMfFRAAI6rjwTCaXn0CS8vrcJQNew=="}' 2026-02-22T20:28:07.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6d9d5f15-2e39-4235-ba79-29e104d4f4a3 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:28:07.521 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:28:07.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:28:07.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDXZptpbJMfFRAAI6rjwTCaXn0CS8vrcJQNew== --osd-uuid 6d9d5f15-2e39-4235-ba79-29e104d4f4a3 2026-02-22T20:28:07.550 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:07.541+0000 7f0a02d61a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:07.551 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:07.545+0000 7f0a02d61a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:07.553 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:07.545+0000 7f0a02d61a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:07.553 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:07.545+0000 7f0a02d61a40 -1 bdev(0x562c7deaec00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:28:07.553 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:07.545+0000 7f0a02d61a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:28:08.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:28:08.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:28:08.198 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:28:08.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:28:08.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:28:08.403 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:28:08.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:28:08.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:28:08.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:28:08.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:28:08.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:28:08.421 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:08.413+0000 7fc9a5a17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:08.422 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:08.417+0000 7fc9a5a17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:08.423 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:08.417+0000 7fc9a5a17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:08.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:28:08.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:28:08.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:28:08.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:28:08.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:28:08.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:08.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:28:08.563 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:28:08.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:08.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:28:08.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:08.804 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:08.797+0000 7fc9a5a17a40 -1 Falling back to public interface 2026-02-22T20:28:09.372 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:09.365+0000 7fc9a5a17a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:28:09.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:09.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:09.725 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:28:09.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:28:09.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:09.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:28:09.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:10.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:10.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:10.890 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:28:10.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:28:10.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:10.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:28:11.048 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2561497719,v1:127.0.0.1:6811/2561497719] [v2:127.0.0.1:6812/2561497719,v1:127.0.0.1:6813/2561497719] exists,up 6d9d5f15-2e39-4235-ba79-29e104d4f4a3 2026-02-22T20:28:11.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:28:11.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:28:11.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:28:11.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:340: auto_repair_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:28:11.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:341: auto_repair_erasure_coded: run_osd td/osd-scrub-repair 2 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:28:11.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:28:11.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:28:11.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:28:11.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:28:11.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:28:11.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:28:11.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:28:11.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:28:11.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:28:11.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:28:11.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:28:11.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:28:11.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:28:11.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:28:11.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:28:11.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:11.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:11.054 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:11.054 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:11.054 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:11.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:11.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:11.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:28:11.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:28:11.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:28:11.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:28:11.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:28:11.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:28:11.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:28:11.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:28:11.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:28:11.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:28:11.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:28:11.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:28:11.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:28:11.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0853fb1c-b636-4144-9e65-7a300d9f363b 2026-02-22T20:28:11.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 0853fb1c-b636-4144-9e65-7a300d9f363b' 2026-02-22T20:28:11.056 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 0853fb1c-b636-4144-9e65-7a300d9f363b 2026-02-22T20:28:11.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:28:11.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDbZptpviauAxAAINgm2L6mEcoy6qD/pyvs+g== 2026-02-22T20:28:11.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDbZptpviauAxAAINgm2L6mEcoy6qD/pyvs+g=="}' 2026-02-22T20:28:11.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0853fb1c-b636-4144-9e65-7a300d9f363b -i td/osd-scrub-repair/2/new.json 2026-02-22T20:28:11.227 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:28:11.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:28:11.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQDbZptpviauAxAAINgm2L6mEcoy6qD/pyvs+g== --osd-uuid 0853fb1c-b636-4144-9e65-7a300d9f363b 2026-02-22T20:28:11.263 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:11.257+0000 7f806b945a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:11.265 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:11.257+0000 7f806b945a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:11.266 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:11.257+0000 7f806b945a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:11.267 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:11.261+0000 7f806b945a40 -1 bdev(0x558b5d1bac00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:28:11.267 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:11.261+0000 7f806b945a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:28:11.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:28:11.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:28:11.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:28:11.942 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:28:11.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:28:12.153 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:28:12.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:28:12.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:28:12.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:28:12.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:28:12.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:28:12.181 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:12.173+0000 7f6e37ce9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:12.183 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:12.177+0000 7f6e37ce9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:12.184 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:12.177+0000 7f6e37ce9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:12.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:28:12.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:28:12.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:28:12.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:28:12.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:28:12.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:12.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:28:12.313 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:28:12.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:12.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:28:12.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:12.592 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:12.585+0000 7f6e37ce9a40 -1 Falling back to public interface 2026-02-22T20:28:13.266 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:13.257+0000 7f6e37ce9a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:28:13.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:13.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:13.475 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:28:13.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:28:13.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:13.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:28:13.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:14.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:14.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:14.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:28:14.637 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:28:14.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:14.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:28:14.792 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/563983577,v1:127.0.0.1:6819/563983577] [v2:127.0.0.1:6820/563983577,v1:127.0.0.1:6821/563983577] exists,up 0853fb1c-b636-4144-9e65-7a300d9f363b 2026-02-22T20:28:14.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:28:14.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:28:14.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:28:14.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:343: auto_repair_erasure_coded: create_rbd_pool 2026-02-22T20:28:14.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:28:14.942 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:28:14.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:28:14.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:28:15.173 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:28:15.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:28:16.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:28:16.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:344: auto_repair_erasure_coded: wait_for_clean 2026-02-22T20:28:16.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:28:16.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:28:16.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:28:16.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:28:16.464 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:28:16.465 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:28:16.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:28:16.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:28:16.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:28:16.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:28:16.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:28:16.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:28:16.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:28:16.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:28:16.562 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:28:16.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:28:16.722 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:28:16.722 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:28:16.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:28:16.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:16.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:28:16.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:28:16.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:28:16.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:28:16.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:16.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:28:16.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-22T20:28:16.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-22T20:28:16.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-22T20:28:16.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:16.867 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:28:16.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-22T20:28:16.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-22T20:28:16.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-60129542146' 2026-02-22T20:28:16.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:16.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:28:16.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:16.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:28:16.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:28:16.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:16.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:28:16.941 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:28:16.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:28:16.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:17.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:28:17.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:28:18.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:28:18.107 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:18.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-22T20:28:18.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:18.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-22T20:28:18.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:18.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:28:18.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-22T20:28:18.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:18.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-22T20:28:18.274 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672963 2026-02-22T20:28:18.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-22T20:28:18.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:28:18.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-22T20:28:18.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:18.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-22T20:28:18.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:18.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:28:18.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-22T20:28:18.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:18.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-22T20:28:18.447 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542146 2026-02-22T20:28:18.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-22T20:28:18.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:28:18.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-22T20:28:18.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:28:18.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:28:18.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:28:18.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:28:18.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:28:18.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:28:18.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:28:18.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:28:18.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:28:18.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:28:18.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:28:18.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:28:18.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:28:18.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:28:18.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:28:19.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:28:19.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:28:19.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:28:19.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:347: auto_repair_erasure_coded: create_ec_pool ecpool true k=2 m=1 2026-02-22T20:28:19.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-22T20:28:19.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:28:19.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-22T20:28:19.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:28:19.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-22T20:28:19.347 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:28:19.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-22T20:28:19.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-22T20:28:19.654 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' created 2026-02-22T20:28:19.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:28:20.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-22T20:28:20.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-22T20:28:20.869 INFO:tasks.workunit.client.0.trial150.stderr:set pool 2 allow_ec_overwrites to true 2026-02-22T20:28:20.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:28:20.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:28:20.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:28:20.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:28:20.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:28:20.884 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:28:20.884 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:28:20.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:28:20.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:28:20.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:28:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:28:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:28:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:28:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:28:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:28:21.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:28:21.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:28:21.159 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:28:21.159 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:28:21.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:28:21.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:21.160 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:28:21.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-22T20:28:21.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-22T20:28:21.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-22T20:28:21.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:21.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:28:21.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-22T20:28:21.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-22T20:28:21.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965' 2026-02-22T20:28:21.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:21.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:28:21.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-22T20:28:21.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-22T20:28:21.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965 2-60129542148' 2026-02-22T20:28:21.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:21.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-22T20:28:21.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:21.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:28:21.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-22T20:28:21.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:21.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-22T20:28:21.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-22T20:28:21.386 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836485 2026-02-22T20:28:21.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:21.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-22T20:28:21.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:28:22.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:28:22.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:22.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-22T20:28:22.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:22.712 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-22T20:28:22.712 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:22.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:28:22.714 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-22T20:28:22.714 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:22.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-22T20:28:22.716 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672965 2026-02-22T20:28:22.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-22T20:28:22.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:28:22.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-22T20:28:22.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:22.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-22T20:28:22.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:22.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:28:22.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-22T20:28:22.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:22.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-22T20:28:22.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-22T20:28:22.879 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542148 2026-02-22T20:28:22.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:28:23.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-22T20:28:23.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:28:23.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:28:23.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:28:23.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:28:23.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:28:23.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:28:23.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:28:23.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:28:23.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:28:23.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:28:23.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:28:23.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:28:23.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:28:23.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:28:23.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:28:23.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:28:23.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:28:23.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:28:23.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:28:23.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:350: auto_repair_erasure_coded: local payload=ABCDEF 2026-02-22T20:28:23.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:351: auto_repair_erasure_coded: echo ABCDEF 2026-02-22T20:28:23.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:352: auto_repair_erasure_coded: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:28:23.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2026-02-22T20:28:23.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2026-02-22T20:28:23.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:28:23.590 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2026-02-22T20:28:23.590 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-22T20:28:23.590 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:28:23.590 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:28:23.590 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:28:23.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2026-02-22T20:28:23.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:28:23.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2026-02-22T20:28:23.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:356: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:28:23.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:28:23.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:28:23.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:28:23.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:28:23.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:28:23.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:28:23.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:28:23.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:28:23.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:28:23.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:28:23.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:28:23.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:28:23.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:28:23.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:28:23.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:28:24.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:28:24.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:28:24.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:28:24.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:28:24.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:28:24.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:28:24.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:28:24.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-22T20:28:24.423 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-22T20:28:24.423 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-22T20:28:24.424 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:28:24.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:28:24.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:28:24.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:24.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:24.529 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:24.529 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:24.529 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:24.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:24.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:24.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:28:24.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:28:24.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:28:24.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:28:24.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:28:24.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:28:24.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:28:24.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:28:24.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:28:24.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:28:24.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:28:24.530 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:28:24.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:28:24.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:28:24.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:28:24.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:28:24.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:28:24.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:28:24.554 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:24.545+0000 7fa2da6fba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:24.555 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:24.549+0000 7fa2da6fba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:24.556 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:24.549+0000 7fa2da6fba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:24.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:28:24.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:28:24.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:28:24.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:28:24.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:28:24.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:24.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:28:24.693 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:28:24.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:24.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:28:24.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:24.912 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:24.905+0000 7fa2da6fba40 -1 Falling back to public interface 2026-02-22T20:28:25.498 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:25.489+0000 7fa2da6fba40 -1 osd.1 29 log_to_monitors true 2026-02-22T20:28:25.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:25.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:25.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:28:25.850 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:28:25.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:25.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:28:26.003 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 33 up_thru 33 down_at 30 last_clean_interval [10,29) [v2:127.0.0.1:6810/1840583360,v1:127.0.0.1:6811/1840583360] [v2:127.0.0.1:6812/1840583360,v1:127.0.0.1:6813/1840583360] exists,up 6d9d5f15-2e39-4235-ba79-29e104d4f4a3 2026-02-22T20:28:26.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:28:26.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:28:26.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:28:26.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:28:26.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:28:26.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:28:26.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:28:26.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:28:26.004 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:28:26.004 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:28:26.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:28:26.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:28:26.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:28:26.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:28:26.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:28:26.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:28:26.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:28:26.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:28:26.109 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:28:26.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:28:26.268 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:28:26.268 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:28:26.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:28:26.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:26.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:28:26.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-22T20:28:26.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-22T20:28:26.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-22T20:28:26.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:26.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:28:26.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920770 2026-02-22T20:28:26.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920770 2026-02-22T20:28:26.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-141733920770' 2026-02-22T20:28:26.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:26.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:28:26.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-22T20:28:26.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-22T20:28:26.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-141733920770 2-60129542150' 2026-02-22T20:28:26.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:26.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-22T20:28:26.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:26.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:28:26.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-22T20:28:26.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:26.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-22T20:28:26.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-22T20:28:26.489 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836487 2026-02-22T20:28:26.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:26.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T20:28:26.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:28:27.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:28:27.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:27.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T20:28:27.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:28:28.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:28:28.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:28.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836487 2026-02-22T20:28:28.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:28.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-141733920770 2026-02-22T20:28:28.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:28.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:28:28.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-141733920770 2026-02-22T20:28:28.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:28.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=141733920770 2026-02-22T20:28:28.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 141733920770' 2026-02-22T20:28:28.986 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 141733920770 2026-02-22T20:28:28.987 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:28:29.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 141733920770 -lt 141733920770 2026-02-22T20:28:29.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:29.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-22T20:28:29.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:29.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:28:29.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-22T20:28:29.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:29.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-22T20:28:29.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-22T20:28:29.162 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542150 2026-02-22T20:28:29.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:28:29.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542150 -lt 60129542150 2026-02-22T20:28:29.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:28:29.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:28:29.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:28:29.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:28:29.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:28:29.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:28:29.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:28:29.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:28:29.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:28:29.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:28:29.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:28:29.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:28:29.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:28:29.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:28:29.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:28:29.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:28:29.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:28:29.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:28:29.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: auto_repair_erasure_coded: get_pg ecpool SOMETHING 2026-02-22T20:28:29.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:28:29.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:28:29.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:28:29.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:28:30.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:358: auto_repair_erasure_coded: local pgid=2.0 2026-02-22T20:28:30.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:359: auto_repair_erasure_coded: get_last_scrub_stamp 2.0 2026-02-22T20:28:30.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:28:30.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:28:30.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:28:30.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:28:30.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:359: auto_repair_erasure_coded: wait_for_scrub 2.0 2026-02-22T20:28:27.348556+0000 2026-02-22T20:28:30.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:28:30.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:28:27.348556+0000 2026-02-22T20:28:30.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:28:30.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:28:30.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:28:30.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:28:30.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:28:30.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:28:30.175 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:28:30.175 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:28:30.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:28:27.348556+0000 '>' 2026-02-22T20:28:27.348556+0000 2026-02-22T20:28:30.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:28:31.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:28:31.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:28:31.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:28:31.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:28:31.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:28:31.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:28:31.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:28:31.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:28:27.348556+0000 '>' 2026-02-22T20:28:27.348556+0000 2026-02-22T20:28:31.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:28:32.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:28:32.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:28:32.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:28:32.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:28:32.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:28:32.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:28:32.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:28:32.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:28:27.348556+0000 '>' 2026-02-22T20:28:27.348556+0000 2026-02-22T20:28:32.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:28:33.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:28:33.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:28:33.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:28:33.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:28:33.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:28:33.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:28:33.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:28:33.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:28:27.348556+0000 '>' 2026-02-22T20:28:27.348556+0000 2026-02-22T20:28:33.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:28:34.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:28:34.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:28:34.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:28:34.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:28:34.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:28:34.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:28:34.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:28:34.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:28:27.348556+0000 '>' 2026-02-22T20:28:27.348556+0000 2026-02-22T20:28:34.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:28:35.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:28:35.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:28:35.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:28:35.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:28:35.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:28:35.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:28:35.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:28:36.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:28:34.296289+0000 '>' 2026-02-22T20:28:27.348556+0000 2026-02-22T20:28:36.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:28:36.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:360: auto_repair_erasure_coded: wait_for_clean 2026-02-22T20:28:36.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:28:36.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:28:36.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:28:36.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:28:36.144 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:28:36.145 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:28:36.145 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:28:36.145 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:28:36.145 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:28:36.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:28:36.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:28:36.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:28:36.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:28:36.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:28:36.254 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:28:36.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:28:36.417 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:28:36.417 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:28:36.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:28:36.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:36.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:28:36.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-22T20:28:36.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-22T20:28:36.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490' 2026-02-22T20:28:36.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:36.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:28:36.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920773 2026-02-22T20:28:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920773 2026-02-22T20:28:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-141733920773' 2026-02-22T20:28:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:36.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:28:36.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-22T20:28:36.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-22T20:28:36.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-141733920773 2-60129542153' 2026-02-22T20:28:36.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:36.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-22T20:28:36.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:36.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:28:36.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836490 2026-02-22T20:28:36.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:36.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836490 2026-02-22T20:28:36.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-22T20:28:36.639 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836490 2026-02-22T20:28:36.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:36.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-22T20:28:36.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:28:37.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:28:37.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:37.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-22T20:28:37.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:28:38.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:28:38.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:39.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836490 2026-02-22T20:28:39.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:39.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-141733920773 2026-02-22T20:28:39.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:39.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:28:39.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-141733920773 2026-02-22T20:28:39.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:39.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=141733920773 2026-02-22T20:28:39.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 141733920773' 2026-02-22T20:28:39.130 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 141733920773 2026-02-22T20:28:39.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:28:39.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 141733920773 -lt 141733920773 2026-02-22T20:28:39.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:39.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-22T20:28:39.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:39.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:28:39.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-22T20:28:39.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:39.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-22T20:28:39.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-22T20:28:39.305 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542153 2026-02-22T20:28:39.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:28:39.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542153 2026-02-22T20:28:39.472 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:28:39.473 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:28:39.473 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:28:39.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:28:39.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:28:39.666 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:28:39.666 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:28:39.666 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:28:39.666 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:28:39.666 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:28:39.666 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:28:39.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:28:39.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:28:39.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:28:39.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:28:40.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:28:40.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:28:40.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:28:40.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: auto_repair_erasure_coded: get_not_primary ecpool SOMETHING 2026-02-22T20:28:40.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=ecpool 2026-02-22T20:28:40.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:28:40.013 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary ecpool SOMETHING 2026-02-22T20:28:40.013 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-22T20:28:40.013 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:28:40.013 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:28:40.013 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:28:40.175 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=2 2026-02-22T20:28:40.175 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:28:40.175 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2026-02-22T20:28:40.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:363: auto_repair_erasure_coded: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:28:40.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:28:40.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:28:40.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:28:40.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:28:40.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:28:40.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:28:40.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:28:40.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:28:40.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:28:40.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:28:40.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:28:40.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:28:40.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:28:40.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:28:40.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:28:40.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:28:40.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:28:40.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:28:40.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:28:40.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:28:40.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:28:40.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:28:40.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-22T20:28:40.718 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-22T20:28:40.718 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-22T20:28:40.718 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#-3:00000000:::deep_scrub_1.0:head#, (61) No data available 2026-02-22T20:28:40.718 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#-3:00000000:::scrub_1.0:head#, (61) No data available 2026-02-22T20:28:40.718 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:28:40.718 INFO:tasks.workunit.client.0.trial150.stdout:hinfo_key 2026-02-22T20:28:40.718 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:28:40.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:28:40.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:28:40.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:28:40.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:28:40.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:28:40.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:28:40.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:28:40.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:28:40.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:28:40.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:28:40.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:28:40.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:28:40.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:28:40.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:28:40.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:28:40.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:28:40.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:40.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:40.820 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:40.820 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:40.820 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:40.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:40.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:40.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:28:40.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:28:40.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:28:40.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:28:40.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:28:40.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:28:40.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:28:40.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:28:40.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:28:40.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:28:40.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:28:40.823 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:28:40.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-auto-repair=true --osd-deep-scrub-interval=5 --osd-scrub-max-interval=5 --osd-scrub-min-interval=5 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:28:40.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:28:40.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:28:40.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:28:40.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:28:40.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:28:40.840 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:40.832+0000 7f7f6ddaea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:40.841 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:40.832+0000 7f7f6ddaea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:40.842 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:40.832+0000 7f7f6ddaea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:40.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:28:40.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:28:40.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:28:40.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:28:40.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:28:40.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:40.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:28:40.984 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:28:40.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:40.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:28:41.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:41.217 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:41.208+0000 7f7f6ddaea40 -1 Falling back to public interface 2026-02-22T20:28:41.867 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:41.860+0000 7f7f6ddaea40 -1 osd.1 36 log_to_monitors true 2026-02-22T20:28:42.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:42.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:42.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:28:42.140 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:28:42.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:42.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:28:42.294 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 37 last_clean_interval [33,36) [v2:127.0.0.1:6810/3151943171,v1:127.0.0.1:6811/3151943171] [v2:127.0.0.1:6812/3151943171,v1:127.0.0.1:6813/3151943171] exists,up 6d9d5f15-2e39-4235-ba79-29e104d4f4a3 2026-02-22T20:28:42.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:28:42.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:28:42.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:28:42.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:28:42.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:28:42.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:28:42.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:28:42.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:28:42.295 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:28:42.295 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:28:42.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:28:42.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:28:42.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:28:42.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:28:42.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:28:42.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:28:42.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:28:42.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:28:42.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:28:42.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:28:42.556 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:28:42.556 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:28:42.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:28:42.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:42.556 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:28:42.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-22T20:28:42.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-22T20:28:42.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-22T20:28:42.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:42.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:28:42.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691842 2026-02-22T20:28:42.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691842 2026-02-22T20:28:42.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-171798691842' 2026-02-22T20:28:42.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:28:42.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:28:42.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-22T20:28:42.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-22T20:28:42.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-171798691842 2-60129542155' 2026-02-22T20:28:42.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:42.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-22T20:28:42.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:42.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:28:42.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-22T20:28:42.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:42.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-22T20:28:42.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-22T20:28:42.782 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836493 2026-02-22T20:28:42.782 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:42.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-22T20:28:42.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:28:43.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:28:43.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:44.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-22T20:28:44.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:28:45.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:28:45.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:28:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-22T20:28:45.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:45.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-171798691842 2026-02-22T20:28:45.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:45.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:28:45.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-171798691842 2026-02-22T20:28:45.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:45.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691842 2026-02-22T20:28:45.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 171798691842' 2026-02-22T20:28:45.277 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 171798691842 2026-02-22T20:28:45.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:28:45.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691842 -lt 171798691842 2026-02-22T20:28:45.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:28:45.444 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-22T20:28:45.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:28:45.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:28:45.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-22T20:28:45.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:28:45.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-22T20:28:45.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-22T20:28:45.449 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542155 2026-02-22T20:28:45.449 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:28:45.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-22T20:28:45.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:28:45.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:28:45.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:28:45.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:28:45.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:28:45.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:28:45.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:28:45.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:28:45.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:28:45.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:28:45.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:28:45.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:28:45.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:28:45.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:28:45.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:28:46.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:28:46.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:28:46.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:28:46.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:364: auto_repair_erasure_coded: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:28:46.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:365: auto_repair_erasure_coded: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:28:46.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:28:46.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:28:46.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:28:46.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:28:46.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:28:46.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:28:46.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:28:46.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:28:46.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:28:46.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:28:46.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:28:46.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:28:46.292 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:28:46.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:28:46.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:28:46.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:28:46.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:28:46.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:28:46.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:28:46.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:28:46.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:28:46.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:28:46.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:28:46.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:28:46.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:46.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:46.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:28:46.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:28:46.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:28:46.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:28:46.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:28:46.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:28:46.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:28:46.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:28:46.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:28:46.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:28:46.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:28:46.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:28:46.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:28:46.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:28:46.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:28:46.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:28:46.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:28:46.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:28:46.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:28:46.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:28:46.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:28:46.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:28:46.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:28:46.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:28:46.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:28:46.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:28:46.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:28:46.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:28:46.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:28:46.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:28:46.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:46.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:46.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:28:46.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:28:46.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:28:46.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:28:46.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:28:46.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:46.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:46.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:28:46.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:28:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:28:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:28:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:28:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_appends td/osd-scrub-repair 2026-02-22T20:28:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:874: TEST_corrupt_and_repair_jerasure_appends: corrupt_and_repair_jerasure td/osd-scrub-repair false 2026-02-22T20:28:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2026-02-22T20:28:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: corrupt_and_repair_jerasure: local allow_overwrites=false 2026-02-22T20:28:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: corrupt_and_repair_jerasure: local poolname=ecpool 2026-02-22T20:28:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:861: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2026-02-22T20:28:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:28:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:28:46.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:28:46.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:28:46.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:28:46.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:28:46.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:28:46.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:46.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:46.355 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:46.355 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:46.355 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:46.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:46.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:28:46.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:28:46.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:28:46.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:28:46.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:28:46.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:28:46.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:28:46.383 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:28:46.383 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:28:46.383 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:28:46.384 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:28:46.384 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:46.384 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:46.384 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:28:46.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:28:46.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:28:46.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:28:46.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:28:46.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:28:46.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:28:46.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:28:46.440 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:28:46.440 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:28:46.440 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:28:46.441 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:28:46.441 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:46.441 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:46.441 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:28:46.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:28:46.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:28:46.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:862: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2026-02-22T20:28:46.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:28:46.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:28:46.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:28:46.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:28:46.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:28:46.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:28:46.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:28:46.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:46.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:46.589 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:46.590 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:46.590 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:46.590 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:46.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:28:46.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:28:46.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: seq 0 3 2026-02-22T20:28:46.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-22T20:28:46.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2026-02-22T20:28:46.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:28:46.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:28:46.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:28:46.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:46.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:28:46.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:28:46.613 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:28:46.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e9ff25cb-348d-4750-8022-0e599590132c 2026-02-22T20:28:46.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e9ff25cb-348d-4750-8022-0e599590132c' 2026-02-22T20:28:46.614 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 e9ff25cb-348d-4750-8022-0e599590132c 2026-02-22T20:28:46.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:28:46.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD+Zptp7V0QJRAA5y9ZCHWyQsQ38hR8LtB21g== 2026-02-22T20:28:46.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD+Zptp7V0QJRAA5y9ZCHWyQsQ38hR8LtB21g=="}' 2026-02-22T20:28:46.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e9ff25cb-348d-4750-8022-0e599590132c -i td/osd-scrub-repair/0/new.json 2026-02-22T20:28:46.713 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:28:46.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:28:46.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD+Zptp7V0QJRAA5y9ZCHWyQsQ38hR8LtB21g== --osd-uuid e9ff25cb-348d-4750-8022-0e599590132c 2026-02-22T20:28:46.738 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:46.728+0000 7f2e6103ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:46.740 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:46.732+0000 7f2e6103ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:46.741 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:46.732+0000 7f2e6103ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:46.741 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:46.732+0000 7f2e6103ea40 -1 bdev(0x55f58eb4cc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:28:46.741 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:46.732+0000 7f2e6103ea40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:28:47.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:28:47.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:28:47.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:28:47.486 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:28:47.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:28:47.689 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:28:47.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:28:47.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:28:47.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:28:47.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:28:47.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:28:47.715 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:47.704+0000 7fadaedcba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:47.716 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:47.708+0000 7fadaedcba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:47.717 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:47.708+0000 7fadaedcba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:47.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:28:47.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:28:47.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:28:47.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:28:47.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:28:47.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:47.850 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:28:47.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:28:47.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:47.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:28:48.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:48.029 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:48.020+0000 7fadaedcba40 -1 Falling back to public interface 2026-02-22T20:28:48.553 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:48.544+0000 7fadaedcba40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:28:49.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:49.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:49.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:28:49.011 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:28:49.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:49.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:28:49.168 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/228069596,v1:127.0.0.1:6803/228069596] [v2:127.0.0.1:6804/228069596,v1:127.0.0.1:6805/228069596] exists,up e9ff25cb-348d-4750-8022-0e599590132c 2026-02-22T20:28:49.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:28:49.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:28:49.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:49.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:49.171 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:49.171 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:49.171 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:49.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:49.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:49.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:28:49.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:28:49.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:28:49.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:28:49.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:28:49.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:28:49.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:28:49.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:28:49.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:28:49.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:28:49.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:28:49.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:28:49.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:28:49.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=eb5cf14e-b43a-4503-8928-d179ba9c2fc9 2026-02-22T20:28:49.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 eb5cf14e-b43a-4503-8928-d179ba9c2fc9' 2026-02-22T20:28:49.174 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 eb5cf14e-b43a-4503-8928-d179ba9c2fc9 2026-02-22T20:28:49.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:28:49.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQABZ5tpOb3WChAAQyijmvKdGdo9Pt8eHx6RAg== 2026-02-22T20:28:49.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQABZ5tpOb3WChAAQyijmvKdGdo9Pt8eHx6RAg=="}' 2026-02-22T20:28:49.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new eb5cf14e-b43a-4503-8928-d179ba9c2fc9 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:28:49.351 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:28:49.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:28:49.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQABZ5tpOb3WChAAQyijmvKdGdo9Pt8eHx6RAg== --osd-uuid eb5cf14e-b43a-4503-8928-d179ba9c2fc9 2026-02-22T20:28:49.380 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:49.372+0000 7f4847fa6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:49.381 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:49.372+0000 7f4847fa6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:49.382 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:49.372+0000 7f4847fa6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:49.382 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:49.372+0000 7f4847fa6a40 -1 bdev(0x56489618cc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:28:49.383 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:49.376+0000 7f4847fa6a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:28:50.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:28:50.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:28:50.077 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:28:50.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:28:50.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:28:50.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:28:50.269 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:28:50.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:28:50.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:28:50.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:28:50.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:28:50.284 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:50.276+0000 7f930b97ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:50.286 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:50.276+0000 7f930b97ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:50.287 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:50.276+0000 7f930b97ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:50.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:28:50.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:28:50.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:28:50.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:28:50.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:28:50.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:50.440 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:28:50.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:28:50.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:50.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:28:50.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:50.732 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:50.724+0000 7f930b97ca40 -1 Falling back to public interface 2026-02-22T20:28:51.329 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:51.320+0000 7f930b97ca40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:28:51.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:51.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:51.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:28:51.597 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:28:51.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:51.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:28:51.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:52.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:52.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:52.759 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:28:52.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:28:52.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:52.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:28:52.915 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2625464612,v1:127.0.0.1:6811/2625464612] [v2:127.0.0.1:6812/2625464612,v1:127.0.0.1:6813/2625464612] exists,up eb5cf14e-b43a-4503-8928-d179ba9c2fc9 2026-02-22T20:28:52.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:28:52.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:28:52.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:28:52.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-22T20:28:52.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2026-02-22T20:28:52.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:28:52.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:28:52.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:28:52.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:28:52.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:28:52.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:28:52.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:28:52.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:28:52.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:28:52.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:28:52.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:28:52.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:28:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:28:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:28:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:28:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:28:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:28:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:28:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:28:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:28:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:28:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:28:52.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:28:52.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7a0b8055-bf83-4255-90e9-207a20642d28 2026-02-22T20:28:52.923 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 7a0b8055-bf83-4255-90e9-207a20642d28 2026-02-22T20:28:52.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7a0b8055-bf83-4255-90e9-207a20642d28' 2026-02-22T20:28:52.924 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:28:52.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAEZ5tpF3VgNxAAJovgiN77yLBPz7Ugdbg8Cw== 2026-02-22T20:28:52.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAEZ5tpF3VgNxAAJovgiN77yLBPz7Ugdbg8Cw=="}' 2026-02-22T20:28:52.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7a0b8055-bf83-4255-90e9-207a20642d28 -i td/osd-scrub-repair/2/new.json 2026-02-22T20:28:53.094 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:28:53.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:28:53.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEZ5tpF3VgNxAAJovgiN77yLBPz7Ugdbg8Cw== --osd-uuid 7a0b8055-bf83-4255-90e9-207a20642d28 2026-02-22T20:28:53.121 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:53.112+0000 7f5fbfdbba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:53.123 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:53.116+0000 7f5fbfdbba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:53.124 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:53.116+0000 7f5fbfdbba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:53.124 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:53.116+0000 7f5fbfdbba40 -1 bdev(0x55f4cb5eec00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:28:53.125 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:53.116+0000 7f5fbfdbba40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:28:53.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:28:53.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:28:53.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:28:53.755 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:28:53.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:28:53.961 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:28:53.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:28:53.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:28:53.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:28:53.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:28:53.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:28:53.986 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:53.976+0000 7efcfe28ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:53.988 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:53.980+0000 7efcfe28ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:53.989 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:53.980+0000 7efcfe28ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:54.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:28:54.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:28:54.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:28:54.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:28:54.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:28:54.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:54.118 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:28:54.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:28:54.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:54.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:28:54.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:54.345 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:54.336+0000 7efcfe28ea40 -1 Falling back to public interface 2026-02-22T20:28:54.977 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:54.968+0000 7efcfe28ea40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:28:55.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:55.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:55.281 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:28:55.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:28:55.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:55.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:28:55.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:56.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:56.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:56.439 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:28:56.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:28:56.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:56.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:28:56.597 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4046261578,v1:127.0.0.1:6819/4046261578] [v2:127.0.0.1:6820/4046261578,v1:127.0.0.1:6821/4046261578] exists,up 7a0b8055-bf83-4255-90e9-207a20642d28 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:28:56.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:28:56.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:28:56.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:28:56.599 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:28:56.599 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:28:56.599 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:28:56.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:28:56.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:28:56.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:28:56.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8d85cab8-9cb3-470b-943c-d7af3186d73e 2026-02-22T20:28:56.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 8d85cab8-9cb3-470b-943c-d7af3186d73e' 2026-02-22T20:28:56.602 INFO:tasks.workunit.client.0.trial150.stdout:add osd3 8d85cab8-9cb3-470b-943c-d7af3186d73e 2026-02-22T20:28:56.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:28:56.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAIZ5tpZQZcJBAAIjs4K33EvDS+MV7fGLbHPw== 2026-02-22T20:28:56.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAIZ5tpZQZcJBAAIjs4K33EvDS+MV7fGLbHPw=="}' 2026-02-22T20:28:56.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8d85cab8-9cb3-470b-943c-d7af3186d73e -i td/osd-scrub-repair/3/new.json 2026-02-22T20:28:56.774 INFO:tasks.workunit.client.0.trial150.stdout:3 2026-02-22T20:28:56.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-22T20:28:56.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIZ5tpZQZcJBAAIjs4K33EvDS+MV7fGLbHPw== --osd-uuid 8d85cab8-9cb3-470b-943c-d7af3186d73e 2026-02-22T20:28:56.810 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:56.800+0000 7fc2a823aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:56.811 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:56.804+0000 7fc2a823aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:56.812 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:56.804+0000 7fc2a823aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:56.813 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:56.804+0000 7fc2a823aa40 -1 bdev(0x55ed0ff6cc00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-22T20:28:56.813 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:56.804+0000 7fc2a823aa40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-22T20:28:57.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-22T20:28:57.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:28:57.455 INFO:tasks.workunit.client.0.trial150.stdout:adding osd3 key to auth repository 2026-02-22T20:28:57.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-22T20:28:57.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:28:57.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-22T20:28:57.661 INFO:tasks.workunit.client.0.trial150.stdout:start osd.3 2026-02-22T20:28:57.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:28:57.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:28:57.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:28:57.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:28:57.686 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:57.676+0000 7fc21a7aca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:57.687 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:57.680+0000 7fc21a7aca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:57.688 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:57.680+0000 7fc21a7aca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:28:57.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-22T20:28:57.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:28:57.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:28:57.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:28:57.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:28:57.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:57.815 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:28:57.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:28:57.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:57.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:28:57.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:28:58.069 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:58.060+0000 7fc21a7aca40 -1 Falling back to public interface 2026-02-22T20:28:58.661 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:28:58.652+0000 7fc21a7aca40 -1 osd.3 0 log_to_monitors true 2026-02-22T20:28:58.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:28:58.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:28:58.975 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:28:58.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:28:58.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:28:58.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:28:59.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:29:00.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:29:00.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:00.140 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:29:00.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:29:00.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:00.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:29:00.302 INFO:tasks.workunit.client.0.trial150.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2626509093,v1:127.0.0.1:6827/2626509093] [v2:127.0.0.1:6828/2626509093,v1:127.0.0.1:6829/2626509093] exists,up 8d85cab8-9cb3-470b-943c-d7af3186d73e 2026-02-22T20:29:00.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:29:00.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:29:00.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:29:00.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:866: corrupt_and_repair_jerasure: create_rbd_pool 2026-02-22T20:29:00.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:29:00.453 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:29:00.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:29:00.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:29:00.684 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:29:00.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:29:01.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:29:01.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: corrupt_and_repair_jerasure: wait_for_clean 2026-02-22T20:29:01.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:29:01.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:29:01.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:29:01.976 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:29:01.977 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:29:01.977 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:29:01.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:29:01.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:29:01.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:29:02.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:29:02.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:29:02.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:29:02.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:29:02.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:29:02.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:29:02.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:29:02.252 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:29:02.252 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:29:02.252 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:29:02.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:29:02.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:02.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:29:02.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-22T20:29:02.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-22T20:29:02.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T20:29:02.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:02.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:29:02.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-22T20:29:02.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-22T20:29:02.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2026-02-22T20:29:02.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:02.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:29:02.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509443 2026-02-22T20:29:02.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509443 2026-02-22T20:29:02.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-64424509443' 2026-02-22T20:29:02.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:02.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:29:02.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378626 2026-02-22T20:29:02.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378626 2026-02-22T20:29:02.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-64424509443 3-81604378626' 2026-02-22T20:29:02.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:02.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-22T20:29:02.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:02.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:29:02.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-22T20:29:02.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:02.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-22T20:29:02.545 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836484 2026-02-22T20:29:02.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T20:29:02.546 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:02.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:29:02.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:03.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:29:03.714 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:03.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-22T20:29:03.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:03.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-22T20:29:03.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:03.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:29:03.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-22T20:29:03.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:03.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-22T20:29:03.876 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672963 2026-02-22T20:29:03.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-22T20:29:03.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:04.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672963 2026-02-22T20:29:04.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:04.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509443 2026-02-22T20:29:04.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:04.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:29:04.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509443 2026-02-22T20:29:04.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:04.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509443 2026-02-22T20:29:04.044 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509443 2026-02-22T20:29:04.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2026-02-22T20:29:04.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:29:04.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509443 -lt 64424509443 2026-02-22T20:29:04.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:04.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378626 2026-02-22T20:29:04.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:04.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:29:04.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-81604378626 2026-02-22T20:29:04.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:04.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=81604378626 2026-02-22T20:29:04.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 81604378626' 2026-02-22T20:29:04.214 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.3 seq 81604378626 2026-02-22T20:29:04.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:29:04.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 81604378626 -lt 81604378626 2026-02-22T20:29:04.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:29:04.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:04.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:04.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:29:04.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:29:04.582 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:29:04.582 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:29:04.582 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:29:04.582 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:29:04.582 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:29:04.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:29:04.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:29:04.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:29:04.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:04.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:29:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:29:04.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:29:04.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: corrupt_and_repair_jerasure: create_ec_pool ecpool false k=2 m=2 2026-02-22T20:29:04.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-22T20:29:04.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:29:04.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-22T20:29:04.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:29:04.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2026-02-22T20:29:05.142 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:29:05.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-22T20:29:05.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-22T20:29:05.448 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' created 2026-02-22T20:29:05.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:29:06.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-22T20:29:06.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:29:06.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:29:06.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:29:06.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:29:06.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:29:06.461 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:29:06.461 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:29:06.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:29:06.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:29:06.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:29:06.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:29:06.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:29:06.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:29:06.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:29:06.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:29:06.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:29:06.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:29:06.727 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:29:06.727 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:29:06.727 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:29:06.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:29:06.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:06.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:29:06.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-22T20:29:06.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-22T20:29:06.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-22T20:29:06.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:06.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:29:06.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-22T20:29:06.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-22T20:29:06.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2026-02-22T20:29:06.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:06.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:29:06.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509445 2026-02-22T20:29:06.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509445 2026-02-22T20:29:06.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509445' 2026-02-22T20:29:06.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:06.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:29:07.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378628 2026-02-22T20:29:07.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378628 2026-02-22T20:29:07.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509445 3-81604378628' 2026-02-22T20:29:07.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:07.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-22T20:29:07.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:07.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:29:07.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-22T20:29:07.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:07.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-22T20:29:07.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-22T20:29:07.022 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836486 2026-02-22T20:29:07.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:07.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-22T20:29:07.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:07.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-22T20:29:07.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:07.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:29:07.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-22T20:29:07.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:07.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-22T20:29:07.189 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672965 2026-02-22T20:29:07.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-22T20:29:07.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:07.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-22T20:29:07.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:07.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509445 2026-02-22T20:29:07.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:07.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:29:07.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509445 2026-02-22T20:29:07.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:07.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509445 2026-02-22T20:29:07.358 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509445 2026-02-22T20:29:07.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2026-02-22T20:29:07.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:29:07.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509445 -lt 64424509445 2026-02-22T20:29:07.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:07.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378628 2026-02-22T20:29:07.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:07.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:29:07.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-81604378628 2026-02-22T20:29:07.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:07.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=81604378628 2026-02-22T20:29:07.526 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.3 seq 81604378628 2026-02-22T20:29:07.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 81604378628' 2026-02-22T20:29:07.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:29:07.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 81604378628 -lt 81604378628 2026-02-22T20:29:07.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:29:07.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:07.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:07.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:29:07.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:29:07.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:29:07.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:29:07.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:29:07.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:29:07.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:29:07.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:29:08.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:29:08.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:29:08.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:08.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: corrupt_and_repair_erasure_coded: local poolname=ecpool 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:29:08.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:29:08.448 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:29:08.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:29:08.650 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:29:08.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:29:08.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:29:08.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:29:08.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2026-02-22T20:29:08.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-22T20:29:08.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:29:08.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:29:08.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:29:08.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: local primary=3 2026-02-22T20:29:08.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2026-02-22T20:29:08.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-22T20:29:08.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-22T20:29:08.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: sed -e s/3// 2026-02-22T20:29:08.857 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:29:08.857 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:29:09.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-22T20:29:09.015 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:29:09.015 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:29:09.015 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:29:09.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: local -a osds 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: corrupt_and_repair_erasure_coded: local not_primary_first=1 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: corrupt_and_repair_erasure_coded: local not_primary_second=2 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=3 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:29:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:29:09.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:29:09.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:29:09.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:29:09.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:29:09.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:29:09.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:29:09.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:29:09.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:29:09.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-22T20:29:09.511 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:29:09.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:29:09.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:29:09.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:29:09.601 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:29:09.601 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:29:09.601 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:start osd.3 2026-02-22T20:29:09.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:29:09.603 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:29:09.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:29:09.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:29:09.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:29:09.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:29:09.620 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:09.612+0000 7f1d61f7fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:09.622 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:09.612+0000 7f1d61f7fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:09.623 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:09.612+0000 7f1d61f7fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:09.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:29:09.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:29:09.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:29:09.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:29:09.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:29:09.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:09.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:29:09.766 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:29:09.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:09.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:29:09.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:29:10.045 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:10.036+0000 7f1d61f7fa40 -1 Falling back to public interface 2026-02-22T20:29:10.716 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:10.708+0000 7f1d61f7fa40 -1 osd.3 34 log_to_monitors true 2026-02-22T20:29:10.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:29:10.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:10.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:29:10.927 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:29:10.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:10.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:29:11.081 INFO:tasks.workunit.client.0.trial150.stderr:osd.3 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [19,34) [v2:127.0.0.1:6826/3628998865,v1:127.0.0.1:6827/3628998865] [v2:127.0.0.1:6828/3628998865,v1:127.0.0.1:6829/3628998865] exists,up 8d85cab8-9cb3-470b-943c-d7af3186d73e 2026-02-22T20:29:11.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:29:11.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:29:11.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:29:11.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:29:11.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:29:11.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:29:11.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:29:11.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:29:11.082 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:29:11.083 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:29:11.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:29:11.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:29:11.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:29:11.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:29:11.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:29:11.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:29:11.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:29:11.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:29:11.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:29:11.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:29:11.347 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:29:11.347 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:29:11.347 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:29:11.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:29:11.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:11.348 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:29:11.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-22T20:29:11.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-22T20:29:11.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-22T20:29:11.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:11.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:29:11.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-22T20:29:11.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-22T20:29:11.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2026-02-22T20:29:11.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:11.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:29:11.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509447 2026-02-22T20:29:11.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509447 2026-02-22T20:29:11.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509447' 2026-02-22T20:29:11.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:11.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:29:11.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757250 2026-02-22T20:29:11.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757250 2026-02-22T20:29:11.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-64424509447 3-163208757250' 2026-02-22T20:29:11.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:11.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-22T20:29:11.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:11.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:29:11.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-22T20:29:11.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:11.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-22T20:29:11.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-22T20:29:11.646 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836488 2026-02-22T20:29:11.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:11.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-22T20:29:11.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:12.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:29:12.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:12.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-22T20:29:12.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:13.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:29:13.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:14.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-22T20:29:14.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:14.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-22T20:29:14.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:14.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:29:14.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-22T20:29:14.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:14.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-22T20:29:14.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-22T20:29:14.137 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672967 2026-02-22T20:29:14.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:14.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672967 2026-02-22T20:29:14.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:14.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509447 2026-02-22T20:29:14.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:14.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:29:14.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509447 2026-02-22T20:29:14.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:14.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509447 2026-02-22T20:29:14.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2026-02-22T20:29:14.298 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509447 2026-02-22T20:29:14.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:29:14.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509447 -lt 64424509447 2026-02-22T20:29:14.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:14.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-163208757250 2026-02-22T20:29:14.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:14.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:29:14.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-163208757250 2026-02-22T20:29:14.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:14.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757250 2026-02-22T20:29:14.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 163208757250' 2026-02-22T20:29:14.469 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 163208757250 2026-02-22T20:29:14.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:29:14.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757250 -lt 163208757250 2026-02-22T20:29:14.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:29:14.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:14.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:14.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:29:14.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:29:14.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:29:14.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:29:14.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:29:14.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:29:14.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:29:14.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:29:14.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:29:14.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:29:14.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:14.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:15.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:29:15.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:29:15.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:29:15.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-22T20:29:15.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:29:15.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:29:15.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:29:15.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:29:15.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-22T20:29:15.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-22T20:29:15.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:29:15.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:29:15.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:15.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:15.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:15.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:15.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:29:05.442843+0000 2026-02-22T20:29:15.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:29:15.630 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:29:15.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:29:05.442843+0000 2026-02-22T20:29:15.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:29:15.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:29:05.442843+0000 2026-02-22T20:29:15.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:29:15.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:29:15.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:15.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:15.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:15.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:15.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:15.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:15.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:05.442843+0000 '>' 2026-02-22T20:29:05.442843+0000 2026-02-22T20:29:15.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:29:16.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:29:16.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:16.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:16.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:16.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:16.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:16.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:16.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:05.442843+0000 '>' 2026-02-22T20:29:05.442843+0000 2026-02-22T20:29:16.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:29:17.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:29:17.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:17.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:17.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:17.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:17.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:17.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:18.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:15.663202+0000 '>' 2026-02-22T20:29:05.442843+0000 2026-02-22T20:29:18.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:29:18.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:29:18.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:29:18.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:29:18.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:29:18.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:29:18.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:29:18.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:29:18.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:29:18.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:29:18.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:29:18.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:29:18.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:29:18.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:29:18.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:29:18.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:29:18.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:29:18.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:29:18.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:29:18.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:29:18.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:29:18.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:29:18.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:29:18.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:29:18.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-22T20:29:18.506 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-22T20:29:18.506 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-22T20:29:18.506 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:29:18.507 INFO:tasks.workunit.client.0.trial150.stdout:hinfo_key 2026-02-22T20:29:18.507 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:29:18.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:29:18.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:29:18.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:29:18.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:29:18.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:29:18.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:29:18.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:29:18.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:29:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:29:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:29:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:29:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:29:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:29:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:29:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:29:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:29:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:29:18.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:29:18.602 INFO:tasks.workunit.client.0.trial150.stderr:start osd.3 2026-02-22T20:29:18.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:29:18.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:29:18.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:29:18.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:29:18.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:29:18.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:29:18.620 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:18.612+0000 7fab515c6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:18.622 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:18.612+0000 7fab515c6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:18.622 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:18.612+0000 7fab515c6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:18.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:29:18.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:29:18.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:29:18.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:29:18.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:29:18.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:18.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:29:18.763 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:29:18.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:18.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:29:18.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:29:19.005 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:18.996+0000 7fab515c6a40 -1 Falling back to public interface 2026-02-22T20:29:19.583 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:19.572+0000 7fab515c6a40 -1 osd.3 39 log_to_monitors true 2026-02-22T20:29:19.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:29:19.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:19.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:29:19.924 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:29:19.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:19.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:29:20.081 INFO:tasks.workunit.client.0.trial150.stderr:osd.3 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [38,39) [v2:127.0.0.1:6826/842997930,v1:127.0.0.1:6827/842997930] [v2:127.0.0.1:6828/842997930,v1:127.0.0.1:6829/842997930] exists,up 8d85cab8-9cb3-470b-943c-d7af3186d73e 2026-02-22T20:29:20.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:29:20.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:29:20.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:29:20.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:29:20.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:29:20.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:29:20.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:29:20.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:29:20.082 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:29:20.082 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:29:20.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:29:20.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:29:20.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:29:20.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:29:20.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:29:20.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:29:20.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:29:20.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:29:20.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:29:20.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:29:20.344 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:29:20.344 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:29:20.344 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:29:20.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:29:20.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:20.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:29:20.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-22T20:29:20.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-22T20:29:20.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T20:29:20.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:20.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:29:20.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672970 2026-02-22T20:29:20.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672970 2026-02-22T20:29:20.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2026-02-22T20:29:20.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:20.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:29:20.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509449 2026-02-22T20:29:20.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509449 2026-02-22T20:29:20.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449' 2026-02-22T20:29:20.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:20.562 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:29:20.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593730 2026-02-22T20:29:20.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593730 2026-02-22T20:29:20.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-184683593730' 2026-02-22T20:29:20.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:20.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-22T20:29:20.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:20.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:29:20.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-22T20:29:20.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:20.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-22T20:29:20.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T20:29:20.641 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836491 2026-02-22T20:29:20.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:20.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-22T20:29:20.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:21.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:29:21.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:21.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-22T20:29:21.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:21.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672970 2026-02-22T20:29:21.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:21.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:29:21.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672970 2026-02-22T20:29:21.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:21.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672970 2026-02-22T20:29:21.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2026-02-22T20:29:21.965 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672970 2026-02-22T20:29:21.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:22.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672970 -lt 42949672970 2026-02-22T20:29:22.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:22.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509449 2026-02-22T20:29:22.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:22.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:29:22.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509449 2026-02-22T20:29:22.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:22.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509449 2026-02-22T20:29:22.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2026-02-22T20:29:22.141 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509449 2026-02-22T20:29:22.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:29:22.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509449 -lt 64424509449 2026-02-22T20:29:22.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:22.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-184683593730 2026-02-22T20:29:22.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:22.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:29:22.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-184683593730 2026-02-22T20:29:22.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:22.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593730 2026-02-22T20:29:22.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2026-02-22T20:29:22.312 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 184683593730 2026-02-22T20:29:22.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:29:22.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593730 -lt 184683593730 2026-02-22T20:29:22.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:29:22.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:22.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:22.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:29:22.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:29:22.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:29:22.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:29:22.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:29:22.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:29:22.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:29:22.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:29:22.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:29:22.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:29:22.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:22.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:23.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:29:23.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:29:23.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:29:23.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:29:23.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:29:23.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:321: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2026-02-22T20:29:23.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-22T20:29:23.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-22T20:29:23.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=1 2026-02-22T20:29:23.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:29:23.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:29:23.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:29:23.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:29:23.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:29:23.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:29:23.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:29:23.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:29:23.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:29:23.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:29:23.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:29:23.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:29:23.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:29:23.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:29:23.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:29:23.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:29:23.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:29:23.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:29:23.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:29:23.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:29:23.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:29:23.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:29:23.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:29:23.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-22T20:29:23.552 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:29:23.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:29:23.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:29:23.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:29:23.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:29:23.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:29:23.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:29:23.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:29:23.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:29:23.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:29:23.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:29:23.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:29:23.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:29:23.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:29:23.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:29:23.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:29:23.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:29:23.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:29:23.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:29:23.661 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:29:23.661 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:29:23.661 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:29:23.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:23.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:23.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:29:23.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:29:23.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:29:23.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:29:23.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:29:23.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:29:23.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:29:23.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:29:23.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:29:23.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:29:23.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:29:23.662 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:29:23.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:29:23.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:29:23.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:29:23.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:29:23.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:29:23.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:29:23.679 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:23.668+0000 7f848505aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:23.680 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:23.672+0000 7f848505aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:23.681 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:23.672+0000 7f848505aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:23.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:29:23.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:29:23.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:29:23.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:29:23.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:29:23.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:23.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:29:23.818 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:29:23.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:23.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:29:23.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:29:24.065 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:24.056+0000 7f848505aa40 -1 Falling back to public interface 2026-02-22T20:29:24.629 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:24.620+0000 7f848505aa40 -1 osd.1 45 log_to_monitors true 2026-02-22T20:29:24.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:29:24.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:24.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:29:24.977 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:29:24.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:24.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:29:25.138 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 46 last_clean_interval [10,45) [v2:127.0.0.1:6810/1187637787,v1:127.0.0.1:6811/1187637787] [v2:127.0.0.1:6812/1187637787,v1:127.0.0.1:6813/1187637787] exists,up eb5cf14e-b43a-4503-8928-d179ba9c2fc9 2026-02-22T20:29:25.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:29:25.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:29:25.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:29:25.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:29:25.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:29:25.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:29:25.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:29:25.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:29:25.139 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:29:25.140 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:29:25.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:29:25.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:29:25.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:29:25.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:29:25.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:29:25.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:29:25.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:29:25.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:29:25.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:29:25.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:29:25.397 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:29:25.397 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:29:25.397 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:29:25.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:29:25.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:25.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:29:25.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-22T20:29:25.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-22T20:29:25.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-22T20:29:25.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:25.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:29:25.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397506 2026-02-22T20:29:25.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397506 2026-02-22T20:29:25.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-210453397506' 2026-02-22T20:29:25.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:25.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:29:25.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509451 2026-02-22T20:29:25.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509451 2026-02-22T20:29:25.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-210453397506 2-64424509451' 2026-02-22T20:29:25.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:25.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:29:25.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593732 2026-02-22T20:29:25.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593732 2026-02-22T20:29:25.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-210453397506 2-64424509451 3-184683593732' 2026-02-22T20:29:25.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:25.686 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-22T20:29:25.686 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:25.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:29:25.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-22T20:29:25.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:25.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-22T20:29:25.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-22T20:29:25.690 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836493 2026-02-22T20:29:25.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:25.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-22T20:29:25.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:26.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:29:26.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:27.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-22T20:29:27.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:28.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:29:28.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:28.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-22T20:29:28.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:28.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-210453397506 2026-02-22T20:29:28.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:28.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:29:28.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-210453397506 2026-02-22T20:29:28.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:28.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=210453397506 2026-02-22T20:29:28.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 210453397506' 2026-02-22T20:29:28.178 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 210453397506 2026-02-22T20:29:28.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:28.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 210453397506 -lt 210453397506 2026-02-22T20:29:28.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:28.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509451 2026-02-22T20:29:28.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:28.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:29:28.352 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509451 2026-02-22T20:29:28.352 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:28.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509451 2026-02-22T20:29:28.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2026-02-22T20:29:28.353 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509451 2026-02-22T20:29:28.353 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:29:28.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509452 -lt 64424509451 2026-02-22T20:29:28.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:28.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-184683593732 2026-02-22T20:29:28.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:28.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:29:28.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-184683593732 2026-02-22T20:29:28.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:28.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593732 2026-02-22T20:29:28.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 184683593732' 2026-02-22T20:29:28.525 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 184683593732 2026-02-22T20:29:28.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:29:28.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593732 -lt 184683593732 2026-02-22T20:29:28.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:29:28.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:28.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:28.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:29:28.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:29:28.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:29:28.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:29:28.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:29:28.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:29:28.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:29:28.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:29:29.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:29:29.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:29:29.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:29.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:29.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:29:29.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:29:29.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:29:29.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-22T20:29:29.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:29:29.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:29:29.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:29:29.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:29:29.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-22T20:29:29.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-22T20:29:29.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:29:29.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:29:29.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:29.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:29.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:29.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:29.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:29:15.663202+0000 2026-02-22T20:29:29.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:29:29.702 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:29:29.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:29:15.663202+0000 2026-02-22T20:29:29.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:29:29.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:29:15.663202+0000 2026-02-22T20:29:29.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:29:29.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:29:29.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:29.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:29.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:29.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:29.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:29.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:29.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:15.663202+0000 '>' 2026-02-22T20:29:15.663202+0000 2026-02-22T20:29:29.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:29:30.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:29:30.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:30.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:30.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:30.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:30.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:30.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:31.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:15.663202+0000 '>' 2026-02-22T20:29:15.663202+0000 2026-02-22T20:29:31.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:29:32.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:29:32.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:32.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:32.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:32.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:32.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:32.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:32.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:15.663202+0000 '>' 2026-02-22T20:29:15.663202+0000 2026-02-22T20:29:32.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:29:33.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:29:33.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:33.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:33.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:33.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:33.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:33.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:33.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:15.663202+0000 '>' 2026-02-22T20:29:15.663202+0000 2026-02-22T20:29:33.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:29:34.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:29:34.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:34.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:34.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:34.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:34.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:34.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:34.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:15.663202+0000 '>' 2026-02-22T20:29:15.663202+0000 2026-02-22T20:29:34.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:29:35.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:29:35.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:35.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:35.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:35.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:35.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:35.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:35.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:30.439146+0000 '>' 2026-02-22T20:29:15.663202+0000 2026-02-22T20:29:35.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:29:35.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:29:35.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:29:35.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:29:35.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:29:35.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:29:35.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:29:35.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:29:35.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:29:35.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:29:35.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:29:35.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:29:35.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:29:35.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:29:35.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:29:35.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:29:35.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:29:35.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:29:35.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:29:35.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:29:35.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:29:35.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:29:35.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:29:35.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:29:35.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-22T20:29:36.055 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:29:36.055 INFO:tasks.workunit.client.0.trial150.stdout:hinfo_key 2026-02-22T20:29:36.055 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:29:36.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:29:36.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:29:36.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:29:36.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:29:36.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:29:36.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:29:36.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:29:36.155 INFO:tasks.workunit.client.0.trial150.stderr:start osd.3 2026-02-22T20:29:36.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:29:36.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:29:36.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:29:36.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:29:36.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:29:36.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:29:36.174 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:36.164+0000 7f6dcd6f6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:36.176 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:36.168+0000 7f6dcd6f6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:36.177 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:36.168+0000 7f6dcd6f6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:36.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:29:36.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:29:36.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:29:36.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:29:36.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:29:36.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:36.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:29:36.317 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:29:36.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:36.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:29:36.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:29:36.530 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:36.520+0000 7f6dcd6f6a40 -1 Falling back to public interface 2026-02-22T20:29:37.166 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:29:37.160+0000 7f6dcd6f6a40 -1 osd.3 51 log_to_monitors true 2026-02-22T20:29:37.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:29:37.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:37.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:29:37.477 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:29:37.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:37.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:29:37.636 INFO:tasks.workunit.client.0.trial150.stderr:osd.3 up in weight 1 up_from 54 up_thru 54 down_at 51 last_clean_interval [43,51) [v2:127.0.0.1:6826/2175513613,v1:127.0.0.1:6827/2175513613] [v2:127.0.0.1:6828/2175513613,v1:127.0.0.1:6829/2175513613] exists,up 8d85cab8-9cb3-470b-943c-d7af3186d73e 2026-02-22T20:29:37.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:29:37.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:29:37.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:29:37.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:29:37.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:29:37.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:29:37.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:29:37.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:29:37.638 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:29:37.638 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:29:37.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:29:37.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:29:37.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:29:37.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:29:37.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:29:37.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:29:37.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:29:37.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:29:37.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:29:37.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:29:37.913 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:29:37.913 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:29:37.913 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:29:37.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:29:37.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:37.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:29:37.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836496 2026-02-22T20:29:37.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836496 2026-02-22T20:29:37.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496' 2026-02-22T20:29:37.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:37.989 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:29:38.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397509 2026-02-22T20:29:38.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397509 2026-02-22T20:29:38.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-210453397509' 2026-02-22T20:29:38.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:38.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:29:38.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509455 2026-02-22T20:29:38.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509455 2026-02-22T20:29:38.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-210453397509 2-64424509455' 2026-02-22T20:29:38.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:38.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:29:38.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=231928233986 2026-02-22T20:29:38.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 231928233986 2026-02-22T20:29:38.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-210453397509 2-64424509455 3-231928233986' 2026-02-22T20:29:38.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:38.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836496 2026-02-22T20:29:38.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:38.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:29:38.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836496 2026-02-22T20:29:38.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:38.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836496 2026-02-22T20:29:38.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-22T20:29:38.210 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836496 2026-02-22T20:29:38.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:38.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836496 2026-02-22T20:29:38.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:39.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:29:39.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:39.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836496 2026-02-22T20:29:39.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:39.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-210453397509 2026-02-22T20:29:39.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:39.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:29:39.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-210453397509 2026-02-22T20:29:39.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:39.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=210453397509 2026-02-22T20:29:39.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 210453397509' 2026-02-22T20:29:39.543 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 210453397509 2026-02-22T20:29:39.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:39.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 210453397509 -lt 210453397509 2026-02-22T20:29:39.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:39.714 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509455 2026-02-22T20:29:39.714 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:39.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:29:39.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509455 2026-02-22T20:29:39.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:39.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509455 2026-02-22T20:29:39.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2026-02-22T20:29:39.718 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509455 2026-02-22T20:29:39.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:29:39.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509455 -lt 64424509455 2026-02-22T20:29:39.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:39.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-231928233986 2026-02-22T20:29:39.886 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:39.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:29:39.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-231928233986 2026-02-22T20:29:39.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:39.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=231928233986 2026-02-22T20:29:39.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 231928233986' 2026-02-22T20:29:39.889 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 231928233986 2026-02-22T20:29:39.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:29:40.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 231928233986 -lt 231928233986 2026-02-22T20:29:40.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:29:40.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:40.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:40.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:29:40.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:29:40.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:29:40.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:29:40.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:29:40.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:29:40.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:29:40.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:29:40.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:29:40.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:29:40.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:40.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:40.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:29:40.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:29:40.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:29:40.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:29:40.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:29:40.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2026-02-22T20:29:40.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-22T20:29:40.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-22T20:29:40.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=1 2026-02-22T20:29:40.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=2 2026-02-22T20:29:40.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-22T20:29:40.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:29:40.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:29:40.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 133137"' 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 133137' 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 133138"' 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 133138' 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 133137 133138' 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 133137 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/133140: /' 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:29:40.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/133142: /' 2026-02-22T20:29:41.236 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:29:41.236 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:29:41.236 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:29:41.237 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:29:41.237 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:29:41.237 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:29:41.237 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:29:41.237 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:29:41.237 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:29:41.237 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:29:41.237 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:29:41.238 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:29:41.238 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:29:41.238 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:29:41.238 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:29:41.238 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:29:41.238 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:29:41.238 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:29:41.238 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:29:41.238 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:29:41.239 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:29:41.239 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:29:41.239 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-22T20:29:41.239 INFO:tasks.workunit.client.0.trial150.stderr:133140: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:29:41.239 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:29:41.239 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:29:41.239 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:29:41.239 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:29:41.239 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:29:41.239 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:29:41.239 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:29:41.239 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:29:41.240 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:29:41.240 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:29:41.240 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:29:41.240 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:29:41.240 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:29:41.241 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:29:41.241 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:29:41.241 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:29:41.241 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:29:41.241 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:29:41.241 INFO:tasks.workunit.client.0.trial150.stderr:133140: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:29:41.241 INFO:tasks.workunit.client.0.trial150.stderr:133140: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:29:41.241 INFO:tasks.workunit.client.0.trial150.stderr:133140: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:29:41.241 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:41.241 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: start osd.1 2026-02-22T20:29:41.242 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:29:41.276 INFO:tasks.workunit.client.0.trial150.stderr:1133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:29:41.277 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:29:41.277 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T20:29:41.277 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:29:41.277 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:29:41.277 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:29:41.277 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:29:41.277 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T20:29:41.277 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:29:41.278 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-22T20:29:41.278 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:29:41.278 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:29:41.278 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:29:41.278 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:29:41.278 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:29:41.278 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:29:41.278 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:29:41.278 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:29:41.278 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:29:41.279 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T20:29:41.279 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:29:41.279 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:29:41.279 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2026-02-22T20:29:41.279 INFO:tasks.workunit.client.0.trial150.stderr:133142: remove 2#2:eb822e21:::SOMETHING:head# 2026-02-22T20:29:41.279 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-22T20:29:41.279 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:29:41.279 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:29:41.279 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:29:41.279 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:29:41.279 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:29:41.280 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:29:41.280 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:29:41.280 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:29:41.280 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:29:41.280 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:29:41.281 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:29:41.281 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:29:41.281 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:29:41.281 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:29:41.281 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:29:41.281 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:29:41.281 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:29:41.281 INFO:tasks.workunit.client.0.trial150.stderr:133142: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:29:41.281 INFO:tasks.workunit.client.0.trial150.stderr:133142: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:29:41.281 INFO:tasks.workunit.client.0.trial150.stderr:133142: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:29:41.282 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:41.282 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:41.282 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:29:41.282 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:29:41.282 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:29:41.282 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:29:41.282 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:29:41.282 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:29:41.282 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:29:41.282 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:29:41.282 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:29:41.283 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:29:41.283 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:29:41.283 INFO:tasks.workunit.client.0.trial150.stderr:133142: start osd.2 2026-02-22T20:29:41.283 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: 2026-02-22T20:29:41.252+0000 7ff3a04b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: 2026-02-22T20:29:41.256+0000 7ff3a04b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: 2026-02-22T20:29:41.256+0000 7ff3a04b5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: 0 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: 2026-02-22T20:29:41.736+0000 7ff3a04b5a40 -1 Falling back to public interface 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: 2026-02-22T20:29:42.439+0000 7ff3a04b5a40 -1 osd.1 56 log_to_monitors true 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: 1 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: osd.1 up in weight 1 up_from 60 up_thru 60 down_at 57 last_clean_interval [49,56) [v2:127.0.0.1:6810/1858442856,v1:127.0.0.1:6811/1858442856] [v2:127.0.0.1:6812/1858442856,v1:127.0.0.1:6813/1858442856] exists,up eb5cf14e-b43a-4503-8928-d179ba9c2fc9 2026-02-22T20:29:42.731 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:29:42.732 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:29:42.732 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:29:42.732 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:29:42.732 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:29:42.732 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:29:42.732 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:29:42.732 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:29:42.732 INFO:tasks.workunit.client.0.trial150.stderr:133140: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133140: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: g33142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: 2026-02-22T20:29:41.288+0000 7fbabb323a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: 2026-02-22T20:29:41.288+0000 7fbabb323a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: 2026-02-22T20:29:41.292+0000 7fbabb323a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: 0 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: 2026-02-22T20:29:41.780+0000 7fbabb323a40 -1 Falling back to public interface 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: 2026-02-22T20:29:42.503+0000 7fbabb323a40 -1 osd.2 57 log_to_monitors true 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: 1 2026-02-22T20:29:42.759 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:42.760 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:29:42.760 INFO:tasks.workunit.client.0.trial150.stderr:133142: osd.2 up in weight 1 up_from 61 up_thru 46 down_at 57 last_clean_interval [15,57) [v2:127.0.0.1:6818/2757220573,v1:127.0.0.1:6819/2757220573] [v2:127.0.0.1:6820/2757220573,v1:127.0.0.1:6821/2757220573] exists,up 7a0b8055-bf83-4255-90e9-207a20642d28 2026-02-22T20:29:42.760 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:29:42.760 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:29:42.760 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:29:42.760 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:29:42.760 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:29:42.760 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:29:42.760 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:29:42.760 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:29:42.760 INFO:tasks.workunit.client.0.trial150.stderr:133142: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:29:43.288 INFO:tasks.workunit.client.0.trial150.stderr:133142: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:29:43.288 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:29:43.288 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:29:43.288 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:29:43.288 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:29:43.288 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:29:43.288 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:29:43.288 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: 1 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: 2 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: 3' 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836498 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836498 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836498' 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037762 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037762 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836498 1-257698037762' 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=261993005058 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 261993005058 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836498 1-257698037762 2-261993005058' 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=231928233988 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 231928233988 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836498 1-257698037762 2-261993005058 3-231928233988' 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:43.289 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836498 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/et_timeout_delays: echo true 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133142: 1 2026-02-22T20:29:43.306 INFO:tasks.workunit.client.0.trial150.stderr:133142: 2 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: 3' 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037763 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037763 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-257698037763' 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=261993005059 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 261993005059 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-257698037763 2-261993005059' 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=231928233989 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 231928233989 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-257698037763 2-261993005059 3-231928233989' 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:43.307 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-22T20:29:45.953 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:45.953 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:29:45.953 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836498 2026-02-22T20:29:45.953 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:45.953 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836498 2026-02-22T20:29:45.953 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2026-02-22T20:29:45.953 INFO:tasks.workunit.client.0.trial150.stderr:133140: waiting osd.0 seq 21474836498 2026-02-22T20:29:45.954 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:45.954 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836499 -lt 21474836498 2026-02-22T20:29:45.954 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:45.954 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-257698037762 2026-02-22T20:29:45.954 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:45.954 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:29:45.954 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-257698037762 2026-02-22T20:29:45.954 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:45.954 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037762 2026-02-22T20:29:45.954 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 257698037762' 2026-02-22T20:29:45.954 INFO:tasks.workunit.client.0.trial150.stderr:133140: waiting osd.1 seq 257698037762 2026-02-22T20:29:45.954 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:45.955 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 257698037762 2026-02-22T20:29:45.955 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:45.955 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:29:45.955 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:45.955 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 257698037762 2026-02-22T20:29:45.955 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:45.955 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:29:45.955 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:45.955 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037763 -lt 257698037762 2026-02-22T20:29:45.955 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:45.955 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-261993005058 2026-02-22T20:29:45.955 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:45.956 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:29:45.956 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-261993005058 2026-02-22T20:29:45.956 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:45.956 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=261993005058 2026-02-22T20:29:45.956 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 261993005058' 2026-02-22T20:29:45.956 INFO:tasks.workunit.client.0.trial150.stderr:133140: waiting osd.2 sequbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:45.956 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: waiting osd.0 seq 21474836499 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836499 -lt 21474836499 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-257698037763 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-257698037763 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037763 2026-02-22T20:29:45.957 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 257698037763' 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: waiting osd.1 seq 257698037763 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 257698037763 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 257698037763 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037763 -lt 257698037763 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-261993005059 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-261993005059 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=261993005059 2026-02-22T20:29:45.958 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 261993005059' 2026-02-22T20:29:46.810 INFO:tasks.workunit.client.0.trial150.stderr:133142: waiting osd.2 seq 261993005059 2026-02-22T20:29:46.810 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:29:46.810 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 261993005059 -lt 261993005059 2026-02-22T20:29:46.810 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:46.810 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-231928233989 2026-02-22T20:29:46.810 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:46.810 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:29:46.810 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-231928233989 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=231928233989 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 231928233989' 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: waiting osd.3 seq 231928233989 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 231928233989 -lt 231928233989 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:133142: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:29:46.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr: 261993005058 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 261993005059 -lt 261993005058 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-231928233988 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-231928233988 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=231928233988 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 231928233988' 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: waiting osd.3 seq 231928233988 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 231928233989 -lt 231928233988 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:29:46.835 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:29:46.836 INFO:tasks.workunit.client.0.trial150.stderr:133140: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:29:46.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:29:46.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:29:46.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 133138 2026-02-22T20:29:46.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:29:46.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:29:46.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:29:46.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-22T20:29:46.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:29:46.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-22T20:29:46.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:29:46.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:29:46.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:29:46.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:29:46.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-22T20:29:46.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-22T20:29:46.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:29:46.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:29:46.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:46.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:46.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:46.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:47.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:29:30.439146+0000 2026-02-22T20:29:47.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:29:47.301 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:29:47.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:29:30.439146+0000 2026-02-22T20:29:47.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:29:47.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:29:30.439146+0000 2026-02-22T20:29:47.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:29:47.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:29:47.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:47.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:47.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:47.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:47.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:47.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:47.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:30.439146+0000 '>' 2026-02-22T20:29:30.439146+0000 2026-02-22T20:29:47.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:29:48.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:29:48.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:48.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:48.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:48.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:48.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:48.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:48.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:30.439146+0000 '>' 2026-02-22T20:29:30.439146+0000 2026-02-22T20:29:48.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:29:49.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:29:49.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:49.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:49.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:49.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:49.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:49.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:49.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:30.439146+0000 '>' 2026-02-22T20:29:30.439146+0000 2026-02-22T20:29:49.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:29:50.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:29:50.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:50.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:50.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:50.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:50.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:50.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:50.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:30.439146+0000 '>' 2026-02-22T20:29:30.439146+0000 2026-02-22T20:29:50.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:29:51.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:29:51.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:51.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:51.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:51.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:51.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:51.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:52.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:30.439146+0000 '>' 2026-02-22T20:29:30.439146+0000 2026-02-22T20:29:52.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:29:53.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:29:53.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:29:53.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:29:53.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:29:53.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:29:53.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:29:53.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:29:53.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:47.982098+0000 '>' 2026-02-22T20:29:30.439146+0000 2026-02-22T20:29:53.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:29:53.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-22T20:29:53.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:29:53.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:29:53.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:29:53.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 136020"' 2026-02-22T20:29:53.286 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 136020' 2026-02-22T20:29:53.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-22T20:29:53.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:29:53.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:29:53.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 136021"' 2026-02-22T20:29:53.287 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 136021' 2026-02-22T20:29:53.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-22T20:29:53.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 136020 136021' 2026-02-22T20:29:53.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:29:53.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:29:53.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 136020 2026-02-22T20:29:53.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:29:53.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/136023: /' 2026-02-22T20:29:53.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-22T20:29:53.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/136025: /' 2026-02-22T20:29:53.799 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: _ 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: hinfo_key 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: snapset 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:29:53.800 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:29:53.801 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:29:53.802 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:29:53.802 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:29:53.802 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:29:53.802 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:29:53.802 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:29:53.802 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:29:53.802 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-22T20:29:53.802 INFO:tasks.workunit.client.0.trial150.stderr:136023: _ 2026-02-22T20:29:53.802 INFO:tasks.workunit.client.0.trial150.stderr:136023: hinfo_key 2026-02-22T20:29:53.802 INFO:tasks.workunit.client.0.trial150.stderr:136023: snapset 2026-02-22T20:29:53.802 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:29:53.802 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:29:53.803 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:29:53.803 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:29:53.803 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:29:53.803 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:29:53.803 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:29:53.803 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:29:53.803 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:29:53.804 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:29:53.804 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:29:53.804 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:29:53.804 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:29:53.804 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:29:53.804 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:29:53.804 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:29:53.804 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:29:53.804 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:29:53.804 INFO:tasks.workunit.client.0.trial150.stderr:136025: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:29:53.804 INFO:tasks.workunit.client.0.trial150.stderr:136025: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:29:53.804 INFO:tasks.workunit.client.0.trial150.stderr:136025: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:29:53.805 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:29:53.806 INFO:tasks.workunit.client.0.trial150.stderr:136025: start osd.2 2026-02-22T20:29:53.806 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recos.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:29:53.806 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:29:53.806 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:29:53.806 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:29:53.806 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:29:53.806 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:29:53.806 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:29:53.806 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:29:53.807 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:29:53.808 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:29:53.808 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:29:53.808 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:29:53.808 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:29:53.808 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:29:53.808 INFO:tasks.workunit.client.0.trial150.stderr:136023: start osd.1 2026-02-22T20:29:55.283 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:29:55.283 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:29:55.284 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:29:55.284 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:29:55.284 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:29:55.284 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:29:55.284 INFO:tasks.workunit.client.0.trial150.stderr:136023: 2026-02-22T20:29:53.811+0000 7ff2517cea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:55.284 INFO:tasks.workunit.client.0.trial150.stderr:136023: 2026-02-22T20:29:53.815+0000 7ff2517cea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:55.284 INFO:tasks.workunit.client.0.trial150.stderr:136023: 2026-02-22T20:29:53.815+0000 7ff2517cea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:55.284 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:29:55.284 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:29:55.284 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:29:55.284 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:29:55.285 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:29:55.285 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:55.285 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:29:55.285 INFO:tasks.workunit.client.0.trial150.stderr:136023: 0 2026-02-22T20:29:55.285 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:55.285 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:29:55.285 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:29:55.285 INFO:tasks.workunit.client.0.trial150.stderr:136023: 2026-02-22T20:29:54.295+0000 7ff2517cea40 -1 Falling back to public interface 2026-02-22T20:29:55.285 INFO:tasks.workunit.client.0.trial150.stderr:136023: 2026-02-22T20:29:54.975+0000 7ff2517cea40 -1 osd.1 63 log_to_monitors true 2026-02-22T20:29:55.285 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:29:55.285 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:55.286 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:29:55.286 INFO:tasks.workunit.client.0.trial150.stderr:136023: 1 2026-02-22T20:29:55.286 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:55.286 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:29:55.286 INFO:tasks.workunit.client.0.trial150.stderr:136023: osd.1 up in weight 1 up_from 68 up_thru 61 down_at 64 last_clean_interval [60,63) [v2:127.0.0.1:6818/2666516822,v1:127.0.0.1:6819/2666516822] [v2:127.0.0.1:6820/2666516822,v1:127.0.0.1:6821/2666516822] exists,up eb5cf14e-b43a-4503-8928-d179ba9c2fc9 2026-02-22T20:29:55.286 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:29:55.286 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:29:55.286 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:29:55.286 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:29:55.286 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:29:55.286 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:29:55.286 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:29:55.287 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:29:55.287 INFO:tasks.workunit.client.0.trial150.stderr:136023: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:29:55.292 INFO:tasks.workunit.client.0.trial150.stderr:136023: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.very_ops 2026-02-22T20:29:55.292 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-22T20:29:55.292 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:29:55.292 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:29:55.292 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:29:55.292 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:29:55.292 INFO:tasks.workunit.client.0.trial150.stderr:136025: 2026-02-22T20:29:53.811+0000 7f45bfe2ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:55.293 INFO:tasks.workunit.client.0.trial150.stderr:136025: 2026-02-22T20:29:53.811+0000 7f45bfe2ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:55.293 INFO:tasks.workunit.client.0.trial150.stderr:136025: 2026-02-22T20:29:53.815+0000 7f45bfe2ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:29:55.293 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:29:55.293 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:29:55.293 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:29:55.293 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:29:55.293 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:29:55.293 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:55.293 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:29:55.293 INFO:tasks.workunit.client.0.trial150.stderr:136025: 0 2026-02-22T20:29:55.293 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:55.293 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: 2026-02-22T20:29:54.275+0000 7f45bfe2ba40 -1 Falling back to public interface 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: 2026-02-22T20:29:54.907+0000 7f45bfe2ba40 -1 osd.2 63 log_to_monitors true 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: 1 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: osd.2 up in weight 1 up_from 67 up_thru 67 down_at 64 last_clean_interval [61,63) [v2:127.0.0.1:6810/648286447,v1:127.0.0.1:6811/648286447] [v2:127.0.0.1:6812/648286447,v1:127.0.0.1:6813/648286447] exists,up 7a0b8055-bf83-4255-90e9-207a20642d28 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:29:55.294 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:29:55.295 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:29:55.295 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:29:55.295 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:29:55.295 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:29:55.295 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:29:55.295 INFO:tasks.workunit.client.0.trial150.stderr:136025: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:29:55.828 INFO:tasks.workunit.client.0.trial150.stderr:136025: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1sh:1593: get_timeout_delays: echo true 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: 1 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: 2 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: 3' 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836503 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836503 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503' 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776130 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776130 2026-02-22T20:29:55.829 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-292057776130' 2026-02-22T20:29:55.830 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:55.830 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:29:55.830 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808834 2026-02-22T20:29:55.830 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808834 2026-02-22T20:29:55.830 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-292057776130 2-287762808834' 2026-02-22T20:29:55.830 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:55.830 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:29:55.830 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=231928233992 2026-02-22T20:29:55.830 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 231928233992 2026-02-22T20:29:55.830 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-292057776130 2-287762808834 3-231928233992' 2026-02-22T20:29:55.830 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:55.830 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836503 2026-02-22T20:29:55.840 INFO:tasks.workunit.client.0.trial150.stderr:13602593: get_timeout_delays: echo true 2026-02-22T20:29:55.840 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:29:55.840 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: 1 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: 2 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: 3' 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836504 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836504 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504' 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:55.841 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776131 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776131 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-292057776131' 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808835 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808835 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-292057776131 2-287762808835' 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=231928233993 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 231928233993 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-292057776131 2-287762808835 3-231928233993' 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:55.842 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836504 2026-02-22T20:29:57.484 INFO:tasks.workunit.client.0.trial150.stderr:136025: /3: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:57.484 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:29:57.484 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836503 2026-02-22T20:29:57.484 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:57.484 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836503 2026-02-22T20:29:57.484 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2026-02-22T20:29:57.484 INFO:tasks.workunit.client.0.trial150.stderr:136023: waiting osd.0 seq 21474836503 2026-02-22T20:29:57.484 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:57.485 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836503 2026-02-22T20:29:57.485 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:57.485 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:29:57.485 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:57.485 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836504 -lt 21474836503 2026-02-22T20:29:57.485 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:57.485 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-292057776130 2026-02-22T20:29:57.485 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:57.485 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:29:57.485 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-292057776130 2026-02-22T20:29:57.485 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:57.485 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776130 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 292057776130' 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: waiting osd.1 seq 292057776130 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776131 -lt 292057776130 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808834 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808834 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808834 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808834' 2026-02-22T20:29:57.486 INFO:tasks.workunit.client.0.trial150.stderr:136023: waiting osd.2 seq 287762808834 2026-02-22T20:29:57.487 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:29:57.487 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808835 -lt 287762808834 2026-02-22T20:29:57.487 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:57.493 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:57.493 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:29:57.493 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836504 2026-02-22T20:29:57.493 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:57.493 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836504 2026-02-22T20:29:57.493 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2026-02-22T20:29:57.493 INFO:tasks.workunit.client.0.trial150.stderr:136025: waiting osd.0 seq 21474836504 2026-02-22T20:29:57.493 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:57.493 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836504 2026-02-22T20:29:57.493 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:29:57.494 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:29:57.494 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:29:57.494 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836504 -lt 21474836504 2026-02-22T20:29:57.494 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:57.494 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-292057776131 2026-02-22T20:29:57.494 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:57.494 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:29:57.494 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-292057776131 2026-02-22T20:29:57.494 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:57.494 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776131 2026-02-22T20:29:57.494 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 292057776131' 2026-02-22T20:29:57.494 INFO:tasks.workunit.client.0.trial150.stderr:136025: waiting osd.1 seq 292057776131 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776131 -lt 292057776131 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808835 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808835 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808835 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808835' 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: waiting osd.2 seq 287762808835 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:29:57.495 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808835 -lt 287762808835 2026-02-22T20:29:57.496 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:29:58.197 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helphelpers.sh:2267: flush_pg_stats: echo 3-231928233992 2026-02-22T20:29:58.197 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:58.197 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:29:58.198 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-231928233992 2026-02-22T20:29:58.198 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:58.198 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=231928233992 2026-02-22T20:29:58.198 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 231928233992' 2026-02-22T20:29:58.198 INFO:tasks.workunit.client.0.trial150.stderr:136023: waiting osd.3 seq 231928233992 2026-02-22T20:29:58.198 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:29:58.198 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 231928233993 -lt 231928233992 2026-02-22T20:29:58.198 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:29:58.198 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:58.198 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:58.198 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:29:58.199 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:29:58.199 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:29:58.199 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:29:58.199 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:29:58.199 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:29:58.199 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:29:58.199 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:29:58.199 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:29:58.199 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:29:58.199 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:58.200 INFO:tasks.workunit.client.0.trial150.stderr:136023: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:58.200 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:29:58.200 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:29:58.200 INFO:tasks.workunit.client.0.trial150.stderr:136023: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:29:58.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:29:58.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:29:58.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 136021 2026-02-22T20:29:58.207 INFO:tasks.workunit.client.0.trial150.stderr:ers.sh:2267: flush_pg_stats: echo 3-231928233993 2026-02-22T20:29:58.207 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:29:58.207 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:29:58.207 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-231928233993 2026-02-22T20:29:58.207 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:29:58.207 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=231928233993 2026-02-22T20:29:58.207 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 231928233993' 2026-02-22T20:29:58.207 INFO:tasks.workunit.client.0.trial150.stderr:136025: waiting osd.3 seq 231928233993 2026-02-22T20:29:58.207 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:29:58.208 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 231928233993 -lt 231928233993 2026-02-22T20:29:58.208 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:29:58.208 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:58.208 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:58.208 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:29:58.208 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:29:58.208 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:29:58.208 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:29:58.208 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:29:58.208 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:29:58.208 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:29:58.209 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:29:58.209 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:29:58.209 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:29:58.209 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:29:58.209 INFO:tasks.workunit.client.0.trial150.stderr:136025: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:29:58.209 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:29:58.209 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:29:58.209 INFO:tasks.workunit.client.0.trial150.stderr:136025: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:29:58.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:29:58.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:29:58.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:29:58.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:29:58.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-22T20:29:58.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:29:58.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:29:58.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:29:58.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2026-02-22T20:29:58.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-22T20:29:58.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-22T20:29:58.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=3 2026-02-22T20:29:58.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=1 2026-02-22T20:29:58.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-22T20:29:58.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:29:58.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:29:58.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:29:58.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 138298"' 2026-02-22T20:29:58.230 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 138298' 2026-02-22T20:29:58.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:29:58.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:29:58.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:29:58.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 138300"' 2026-02-22T20:29:58.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 138300' 2026-02-22T20:29:58.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:29:58.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-22T20:29:58.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 138298 138300' 2026-02-22T20:29:58.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:29:58.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:29:58.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 138298 2026-02-22T20:29:58.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/138301: /' 2026-02-22T20:29:58.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:29:58.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/138303: /' 2026-02-22T20:29:58.844 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:29:58.844 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:29:58.845 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:29:58.845 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:29:58.845 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:29:58.845 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:29:58.845 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:29:58.845 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:29:58.845 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:29:58.845 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:29:58.845 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:29:58.845 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:29:58.846 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:29:58.846 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:29:58.846 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:29:58.846 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:29:58.846 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:29:58.846 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:29:58.846 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:29:58.846 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:29:58.846 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:29:58.846 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:29:58.846 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-22T20:29:58.846 INFO:tasks.workunit.client.0.trial150.stderr:138301: remove 0#2:eb822e21:::SOMETHING:head# 2026-02-22T20:29:58.847 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:29:58.847 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:29:58.847 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:29:58.847 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:29:58.847 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:29:58.847 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:29:58.847 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:29:58.847 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:29:58.848 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:29:58.848 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:29:58.848 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:29:58.848 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:29:58.848 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:29:58.848 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:29:58.848 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:29:58.848 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:29:58.849 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:29:58.849 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:29:58.849 INFO:tasks.workunit.client.0.trial150.stderr:138301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:29:58.849 INFO:tasks.workunit.client.0.trial150.stderr:138301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:29:58.849 INFO:tasks.workunit.client.0.trial150.stderr:138301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:29:58.849 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:58.849 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:58.849 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:29:58.849 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:29:58.849 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:29:58.849 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:29:58.849 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:29:58.850 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:29:58.850 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:29:58.850 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:29:58.850 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:29:58.850 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:29:58.850 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:29:58.850 INFO:tasks.workunit.client.0.trial150.stderr:138301: start osd.3 2026-02-22T20:29:58.850 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:29:58.883 INFO:tasks.workunit.client.0.trial150.stderr:1138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:29:58.884 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:29:58.884 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:29:58.884 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:29:58.884 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:29:58.884 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:29:58.884 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:29:58.885 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:29:58.886 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:29:58.886 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:29:58.886 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-22T20:29:58.886 INFO:tasks.workunit.client.0.trial150.stderr:138303: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:29:58.886 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:29:58.886 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:29:58.886 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:29:58.886 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:29:58.886 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:29:58.886 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:29:58.886 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:29:58.886 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:29:58.887 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:29:58.887 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:29:58.887 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:29:58.887 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:29:58.887 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:29:58.888 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:29:58.889 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:29:58.889 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:29:58.889 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:29:58.889 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:29:58.889 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:29:58.889 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:29:58.889 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:29:58.889 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:29:58.889 INFO:tasks.workunit.client.0.trial150.stderr:138303: start osd.1 2026-02-22T20:29:58.889 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:30:00.320 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: 2026-02-22T20:29:58.859+0000 7f8c4b700a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: 2026-02-22T20:29:58.859+0000 7f8c4b700a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: 2026-02-22T20:29:58.863+0000 7f8c4b700a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: 0 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: 2026-02-22T20:29:59.287+0000 7f8c4b700a40 -1 Falling back to public interface 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: 2026-02-22T20:29:59.935+0000 7f8c4b700a40 -1 osd.3 69 log_to_monitors true 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: 1 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: osd.3 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [54,69) [v2:127.0.0.1:6818/3366924540,v1:127.0.0.1:6819/3366924540] [v2:127.0.0.1:6820/3366924540,v1:127.0.0.1:6821/3366924540] exists,up 8d85cab8-9cb3-470b-943c-d7af3186d73e 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:30:00.321 INFO:tasks.workunit.client.0.trial150.stderr:138301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138301: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: g38303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: 2026-02-22T20:29:58.895+0000 7ffa78245a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: 2026-02-22T20:29:58.895+0000 7ffa78245a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: 2026-02-22T20:29:58.895+0000 7ffa78245a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: 0 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: 2026-02-22T20:29:59.291+0000 7ffa78245a40 -1 Falling back to public interface 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: 2026-02-22T20:29:59.951+0000 7ffa78245a40 -1 osd.1 69 log_to_monitors true 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: 1 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: osd.1 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [68,69) [v2:127.0.0.1:6826/3381055793,v1:127.0.0.1:6827/3381055793] [v2:127.0.0.1:6828/3381055793,v1:127.0.0.1:6829/3381055793] exists,up eb5cf14e-b43a-4503-8928-d179ba9c2fc9 2026-02-22T20:30:00.367 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:30:00.368 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:30:00.368 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:30:00.368 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:30:00.368 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:30:00.368 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:30:00.368 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:30:00.368 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:30:00.368 INFO:tasks.workunit.client.0.trial150.stderr:138303: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:30:00.874 INFO:tasks.workunit.client.0.trial150.stderr:138303: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:30:00.874 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:30:00.874 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:30:00.874 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:30:00.874 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:30:00.875 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:30:00.875 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:30:00.875 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:30:00.875 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:30:00.875 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:30:00.875 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:30:00.875 INFO:tasks.workunit.client.0.trial150.stderr:138301: 1 2026-02-22T20:30:00.875 INFO:tasks.workunit.client.0.trial150.stderr:138301: 2 2026-02-22T20:30:00.875 INFO:tasks.workunit.client.0.trial150.stderr:138301: 3' 2026-02-22T20:30:00.875 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:30:00.875 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:00.875 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:30:00.876 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836506 2026-02-22T20:30:00.876 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836506 2026-02-22T20:30:00.876 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506' 2026-02-22T20:30:00.876 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:00.876 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:30:00.876 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612610 2026-02-22T20:30:00.876 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612610 2026-02-22T20:30:00.876 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-313532612610' 2026-02-22T20:30:00.876 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:00.876 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:30:00.876 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808837 2026-02-22T20:30:00.876 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808837 2026-02-22T20:30:00.877 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-313532612610 2-287762808837' 2026-02-22T20:30:00.877 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:00.877 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:30:00.877 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612610 2026-02-22T20:30:00.877 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612610 2026-02-22T20:30:00.877 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-313532612610 2-287762808837 3-313532612610' 2026-02-22T20:30:00.877 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:00.877 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836506 2026-02-22T20:30:00.921 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/et_timeout_delays: echo true 2026-02-22T20:30:00.921 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:30:00.921 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:30:00.921 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:30:00.921 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:30:00.921 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:30:00.921 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: 1 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: 2 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: 3' 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836507 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836507 2026-02-22T20:30:00.922 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507' 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612611 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612611 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-313532612611' 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808838 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808838 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-313532612611 2-287762808838' 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:30:00.923 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612611 2026-02-22T20:30:00.924 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612611 2026-02-22T20:30:00.924 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-313532612611 2-287762808838 3-313532612611' 2026-02-22T20:30:00.924 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:00.924 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836507 2026-02-22T20:30:02.551 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:02.552 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:30:02.552 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836506 2026-02-22T20:30:02.552 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:02.552 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836506 2026-02-22T20:30:02.552 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2026-02-22T20:30:02.552 INFO:tasks.workunit.client.0.trial150.stderr:138301: waiting osd.0 seq 21474836506 2026-02-22T20:30:02.552 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:02.552 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836504 -lt 21474836506 2026-02-22T20:30:02.553 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:30:02.553 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:30:02.553 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:02.553 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836507 -lt 21474836506 2026-02-22T20:30:02.553 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:02.553 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-313532612610 2026-02-22T20:30:02.553 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:02.553 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:30:02.553 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-313532612610 2026-02-22T20:30:02.553 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:02.554 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612610 2026-02-22T20:30:02.554 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 313532612610' 2026-02-22T20:30:02.554 INFO:tasks.workunit.client.0.trial150.stderr:138301: waiting osd.1 seq 313532612610 2026-02-22T20:30:02.554 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:30:02.554 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612611 -lt 313532612610 2026-02-22T20:30:02.554 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:02.554 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808837 2026-02-22T20:30:02.554 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:02.554 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:30:02.554 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808837 2026-02-22T20:30:02.555 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:02.555 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808837 2026-02-22T20:30:02.555 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808837' 2026-02-22T20:30:02.555 INFO:tasks.workunit.client.0.trial150.stderr:138301: waiting osd.2 seq 287762808837 2026-02-22T20:30:02.555 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:30:02.555 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808838 -lt 287762808837 2026-02-22T20:30:02.555 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:02.556 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:02.556 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:30:02.556 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836507 2026-02-22T20:30:02.556 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:02.557 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836507 2026-02-22T20:30:02.557 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2026-02-22T20:30:02.557 INFO:tasks.workunit.client.0.trial150.stderr:138303: waiting osd.0 seq 21474836507 2026-02-22T20:30:02.557 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:02.557 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836504 -lt 21474836507 2026-02-22T20:30:02.557 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:30:02.557 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:30:02.557 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:02.557 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836507 -lt 21474836507 2026-02-22T20:30:02.557 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:02.557 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-313532612611 2026-02-22T20:30:02.558 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:02.558 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:30:02.558 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-313532612611 2026-02-22T20:30:02.558 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:02.558 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612611 2026-02-22T20:30:02.558 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 313532612611' 2026-02-22T20:30:02.558 INFO:tasks.workunit.client.0.trial150.stderr:138303: waiting osd.1 seq 313532612611 2026-02-22T20:30:02.558 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:30:02.558 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612611 -lt 313532612611 2026-02-22T20:30:02.558 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:02.559 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808838 2026-02-22T20:30:02.559 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:02.559 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:30:02.559 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808838 2026-02-22T20:30:02.559 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:02.559 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808838 2026-02-22T20:30:02.559 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808838' 2026-02-22T20:30:02.559 INFO:tasks.workunit.client.0.trial150.stderr:138303: waiting osd.2 seq 287762808838 2026-02-22T20:30:02.559 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:30:02.560 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808838 -lt 287762808838 2026-02-22T20:30:02.560 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:03.264 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-313532612611 2026-02-22T20:30:03.264 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-313532612611 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612611 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 313532612611' 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: waiting osd.3 seq 313532612611 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612611 -lt 313532612611 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:30:03.265 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:138303: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:30:03.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:30:03.269 INFO:tasks.workunit.client.0.trial150.stderr::2267: flush_pg_stats: echo 3-313532612610 2026-02-22T20:30:03.269 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:03.269 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:30:03.269 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-313532612610 2026-02-22T20:30:03.269 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:03.269 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612610 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 313532612610' 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: waiting osd.3 seq 313532612610 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612611 -lt 313532612610 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:30:03.270 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:30:03.271 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:30:03.271 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:30:03.271 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:30:03.271 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:03.271 INFO:tasks.workunit.client.0.trial150.stderr:138301: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:03.271 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:30:03.271 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:30:03.271 INFO:tasks.workunit.client.0.trial150.stderr:138301: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:30:03.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:30:03.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:30:03.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 138300 2026-02-22T20:30:03.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:30:03.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:30:03.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:30:03.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-22T20:30:03.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:30:03.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-22T20:30:03.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:30:03.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:30:03.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:30:03.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:30:03.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-22T20:30:03.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-22T20:30:03.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:30:03.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:30:03.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:30:03.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:30:03.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:30:03.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:30:03.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:29:47.982098+0000 2026-02-22T20:30:03.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:30:03.738 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:30:03.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:29:47.982098+0000 2026-02-22T20:30:03.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:30:03.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:29:47.982098+0000 2026-02-22T20:30:03.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:30:03.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:30:03.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:30:03.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:30:03.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:30:03.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:30:03.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:30:03.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:30:03.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:47.982098+0000 '>' 2026-02-22T20:29:47.982098+0000 2026-02-22T20:30:03.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:30:04.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:30:04.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:30:04.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:30:04.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:30:04.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:30:04.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:30:04.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:30:05.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:29:47.982098+0000 '>' 2026-02-22T20:29:47.982098+0000 2026-02-22T20:30:05.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:30:06.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:30:06.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:30:06.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:30:06.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:30:06.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:30:06.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:30:06.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:30:06.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:04.040566+0000 '>' 2026-02-22T20:29:47.982098+0000 2026-02-22T20:30:06.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:30:06.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-22T20:30:06.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:30:06.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:30:06.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:30:06.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 141060"' 2026-02-22T20:30:06.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 141060' 2026-02-22T20:30:06.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:30:06.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:30:06.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:30:06.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:30:06.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 141062"' 2026-02-22T20:30:06.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 141062' 2026-02-22T20:30:06.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-22T20:30:06.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 141060 141062' 2026-02-22T20:30:06.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:30:06.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:30:06.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 141060 2026-02-22T20:30:06.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/141063: /' 2026-02-22T20:30:06.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:30:06.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/141065: /' 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:30:06.712 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-22T20:30:06.713 INFO:tasks.workunit.client.0.trial150.stderr:141065: _ 2026-02-22T20:30:06.713 INFO:tasks.workunit.client.0.trial150.stderr:141065: hinfo_key 2026-02-22T20:30:06.713 INFO:tasks.workunit.client.0.trial150.stderr:141065: snapset 2026-02-22T20:30:06.713 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:30:06.713 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:30:06.713 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:30:06.713 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:30:06.713 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:30:06.713 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:30:06.713 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:30:06.713 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:30:06.716 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:30:06.716 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:30:06.716 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:30:06.716 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:30:06.716 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:30:06.716 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:30:06.716 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:30:06.716 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:30:06.716 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:30:06.717 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:30:06.717 INFO:tasks.workunit.client.0.trial150.stderr:141065: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:30:06.717 INFO:tasks.workunit.client.0.trial150.stderr:141065: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:06.717 INFO:tasks.workunit.client.0.trial150.stderr:141065: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:06.717 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:06.717 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:06.717 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:30:06.717 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:30:06.717 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:30:06.717 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:30:06.717 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:30:06.717 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:30:06.718 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:30:06.718 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:30:06.718 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:30:06.718 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:30:06.718 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:30:06.718 INFO:tasks.workunit.client.0.trial150.stderr:141065: start osd.1 2026-02-22T20:30:06.736 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:30:06.737 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:30:06.737 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:30:06.737 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:30:06.737 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:30:06.737 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:30:06.737 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:30:06.737 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:30:06.737 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:30:06.737 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:30:06.737 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:30:06.738 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:30:06.738 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:30:06.738 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:30:06.738 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:30:06.738 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:30:06.738 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:30:06.738 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:30:06.738 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:30:06.738 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:30:06.738 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:30:06.738 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:30:06.739 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-22T20:30:06.739 INFO:tasks.workunit.client.0.trial150.stderr:141063: Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-22T20:30:06.739 INFO:tasks.workunit.client.0.trial150.stderr:141063: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-22T20:30:06.739 INFO:tasks.workunit.client.0.trial150.stderr:141063: _ 2026-02-22T20:30:06.739 INFO:tasks.workunit.client.0.trial150.stderr:141063: hinfo_key 2026-02-22T20:30:06.739 INFO:tasks.workunit.client.0.trial150.stderr:141063: snapset 2026-02-22T20:30:06.739 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:30:06.739 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:30:06.739 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:30:06.739 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:30:06.739 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:30:06.739 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:30:06.740 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:30:06.740 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:30:06.740 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:30:06.740 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:30:06.741 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:30:06.741 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:30:06.741 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:30:06.741 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:30:06.741 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:30:06.741 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:30:06.741 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:30:06.741 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:30:06.741 INFO:tasks.workunit.client.0.trial150.stderr:141063: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:30:06.741 INFO:tasks.workunit.client.0.trial150.stderr:141063: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:06.742 INFO:tasks.workunit.client.0.trial150.stderr:141063: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:06.742 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:06.742 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:06.742 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:30:06.742 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:30:06.742 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:30:06.742 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:30:06.742 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:30:06.742 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:30:06.742 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:30:06.742 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:30:06.742 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:30:06.743 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:30:06.743 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:30:06.743 INFO:tasks.workunit.client.0.trial150.stderr:141063: start osd.3 2026-02-22T20:30:08.198 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.very_ops 2026-02-22T20:30:08.198 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:30:08.198 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:30:08.198 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:30:08.198 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:30:08.198 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:30:08.198 INFO:tasks.workunit.client.0.trial150.stderr:141065: 2026-02-22T20:30:06.727+0000 7fd722189a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:08.198 INFO:tasks.workunit.client.0.trial150.stderr:141065: 2026-02-22T20:30:06.727+0000 7fd722189a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:08.198 INFO:tasks.workunit.client.0.trial150.stderr:141065: 2026-02-22T20:30:06.731+0000 7fd722189a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:08.198 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:30:08.198 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:30:08.198 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: 0 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: 2026-02-22T20:30:07.139+0000 7fd722189a40 -1 Falling back to public interface 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: 2026-02-22T20:30:07.819+0000 7fd722189a40 -1 osd.1 75 log_to_monitors true 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: 1 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: osd.1 up in weight 1 up_from 79 up_thru 79 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6818/1848348283,v1:127.0.0.1:6819/1848348283] [v2:127.0.0.1:6820/1848348283,v1:127.0.0.1:6821/1848348283] exists,up eb5cf14e-b43a-4503-8928-d179ba9c2fc9 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:30:08.199 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:30:08.200 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:30:08.200 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:30:08.200 INFO:tasks.workunit.client.0.trial150.stderr:141065: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:30:08.219 INFO:tasks.workunit.client.0.trial150.stderr:141065: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:30:08.219 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:30:08.219 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:30:08.219 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: 2026-02-22T20:30:06.747+0000 7f9ce4c03a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: 2026-02-22T20:30:06.747+0000 7f9ce4c03a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: 2026-02-22T20:30:06.751+0000 7f9ce4c03a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: 0 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: 2026-02-22T20:30:07.191+0000 7f9ce4c03a40 -1 Falling back to public interface 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: 2026-02-22T20:30:07.931+0000 7f9ce4c03a40 -1 osd.3 75 log_to_monitors true 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: 1 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: osd.3 up in weight 1 up_from 80 up_thru 80 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6826/3424840830,v1:127.0.0.1:6827/3424840830] [v2:127.0.0.1:6828/3424840830,v1:127.0.0.1:6829/3424840830] exists,up 8d85cab8-9cb3-470b-943c-d7af3186d73e 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:30:08.220 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delsh:1593: get_timeout_delays: echo true 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: 1 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: 2 2026-02-22T20:30:08.745 INFO:tasks.workunit.client.0.trial150.stderr:141065: 3' 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836509 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836509 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509' 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=339302416386 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 339302416386 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-339302416386' 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808840 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808840 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-339302416386 2-287762808840' 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383682 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383682 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-339302416386 2-287762808840 3-343597383682' 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:08.746 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836509 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:14106ays 90 .1 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: 1 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: 2 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: 3' 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836510 2026-02-22T20:30:08.778 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836510 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510' 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=339302416387 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 339302416387 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-339302416387' 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808841 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808841 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-339302416387 2-287762808841' 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383683 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383683 2026-02-22T20:30:08.779 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-339302416387 2-287762808841 3-343597383683' 2026-02-22T20:30:10.420 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest5: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:10.420 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:30:10.420 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836509 2026-02-22T20:30:10.420 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:10.420 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836509 2026-02-22T20:30:10.420 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2026-02-22T20:30:10.420 INFO:tasks.workunit.client.0.trial150.stderr:141065: waiting osd.0 seq 21474836509 2026-02-22T20:30:10.420 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:10.420 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836509 2026-02-22T20:30:10.421 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:30:10.421 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:30:10.421 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:10.421 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836509 2026-02-22T20:30:10.421 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:10.421 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-339302416386 2026-02-22T20:30:10.421 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:10.421 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:30:10.421 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-339302416386 2026-02-22T20:30:10.421 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:10.421 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=339302416386 2026-02-22T20:30:10.421 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 339302416386' 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: waiting osd.1 seq 339302416386 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 339302416387 -lt 339302416386 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808840 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808840 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808840 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808840' 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: waiting osd.2 seq 287762808840 2026-02-22T20:30:10.422 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:30:10.423 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808841 -lt 287762808840 2026-02-22T20:30:10.423 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:10.432 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:10.433 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836510 2026-02-22T20:30:10.433 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:10.433 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:30:10.433 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836510 2026-02-22T20:30:10.433 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:10.433 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836510 2026-02-22T20:30:10.433 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2026-02-22T20:30:10.433 INFO:tasks.workunit.client.0.trial150.stderr:141063: waiting osd.0 seq 21474836510 2026-02-22T20:30:10.433 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:10.433 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836510 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836510 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-339302416387 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-339302416387 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=339302416387 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 339302416387' 2026-02-22T20:30:10.434 INFO:tasks.workunit.client.0.trial150.stderr:141063: waiting osd.1 seq 339302416387 2026-02-22T20:30:10.435 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:30:10.435 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 339302416387 -lt 339302416387 2026-02-22T20:30:10.435 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:10.435 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-287762808841 2026-02-22T20:30:10.435 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:10.435 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:30:10.435 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-287762808841 2026-02-22T20:30:10.435 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:10.435 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808841 2026-02-22T20:30:10.435 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 287762808841' 2026-02-22T20:30:10.435 INFO:tasks.workunit.client.0.trial150.stderr:141063: waiting osd.2 seq 287762808841 2026-02-22T20:30:10.435 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:30:11.131 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 2helpers.sh:2267: flush_pg_stats: echo 3-343597383682 2026-02-22T20:30:11.131 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-343597383682 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383682 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 343597383682' 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: waiting osd.3 seq 343597383682 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383682 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:30:11.132 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:30:11.133 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:30:11.133 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:30:11.133 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:11.133 INFO:tasks.workunit.client.0.trial150.stderr:141065: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:11.133 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:30:11.133 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:30:11.133 INFO:tasks.workunit.client.0.trial150.stderr:141065: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:30:11.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:87762808841 -lt 287762808841 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-343597383683 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-343597383683 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383683 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 343597383683' 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:141063: waiting osd.3 seq 343597383683 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383683 2026-02-22T20:30:11.149 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:30:11.150 INFO:tasks.workunit.client.0.trial150.stderr:141063: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:30:11.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:30:11.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:30:11.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 141062 2026-02-22T20:30:11.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:30:11.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:30:11.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:30:11.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-22T20:30:11.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:30:11.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:30:11.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:30:11.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:30:11.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:30:11.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:30:11.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:30:11.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:30:11.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:30:11.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:30:11.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:30:11.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:30:11.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:30:11.482 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:30:11.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:30:11.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:30:11.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:30:11.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:30:11.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:30:11.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:30:11.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:30:11.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:30:11.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:30:11.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:30:11.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:30:11.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:30:11.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:30:11.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:11.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:11.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:30:11.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:30:11.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:30:11.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:30:11.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:30:11.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:30:11.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:30:11.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:30:11.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:30:11.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:30:11.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:30:11.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:30:11.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:30:11.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:30:11.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:30:11.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:30:11.520 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:30:11.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:30:11.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:30:11.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:30:11.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:30:11.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:30:11.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:30:11.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:30:11.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:30:11.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:30:11.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:30:11.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:30:11.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:30:11.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:30:11.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:11.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:11.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:30:11.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:30:11.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:30:11.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:30:11.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:30:11.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:11.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:11.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:30:11.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:30:11.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:30:11.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:30:11.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:30:11.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_jerasure_overwrites td/osd-scrub-repair 2026-02-22T20:30:11.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:878: TEST_corrupt_and_repair_jerasure_overwrites: '[' true = true ']' 2026-02-22T20:30:11.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:879: TEST_corrupt_and_repair_jerasure_overwrites: corrupt_and_repair_jerasure td/osd-scrub-repair true 2026-02-22T20:30:11.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:857: corrupt_and_repair_jerasure: local dir=td/osd-scrub-repair 2026-02-22T20:30:11.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:858: corrupt_and_repair_jerasure: local allow_overwrites=true 2026-02-22T20:30:11.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:859: corrupt_and_repair_jerasure: local poolname=ecpool 2026-02-22T20:30:11.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:861: corrupt_and_repair_jerasure: run_mon td/osd-scrub-repair a 2026-02-22T20:30:11.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:30:11.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:30:11.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:30:11.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:30:11.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:30:11.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:30:11.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:30:11.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:30:11.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:30:11.557 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:30:11.557 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:11.557 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:11.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:11.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:30:11.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:30:11.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:30:11.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:30:11.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:30:11.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:30:11.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:30:11.590 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:30:11.591 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:30:11.591 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:30:11.591 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:30:11.591 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:11.591 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:11.591 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:30:11.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:30:11.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:30:11.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:30:11.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:30:11.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:30:11.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:30:11.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:30:11.646 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:30:11.647 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:30:11.647 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:30:11.648 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:30:11.648 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:11.648 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:11.648 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:30:11.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:30:11.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:30:11.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:862: corrupt_and_repair_jerasure: run_mgr td/osd-scrub-repair x 2026-02-22T20:30:11.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:30:11.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:30:11.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:30:11.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:30:11.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:30:11.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:30:11.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:30:11.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:30:11.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:30:11.796 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:30:11.796 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:11.796 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:11.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:11.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:30:11.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:30:11.816 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: seq 0 3 2026-02-22T20:30:11.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-22T20:30:11.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 0 2026-02-22T20:30:11.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:30:11.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:30:11.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:30:11.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:30:11.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:30:11.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:30:11.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:30:11.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:30:11.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:30:11.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:30:11.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:30:11.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:30:11.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:30:11.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:30:11.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:30:11.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:30:11.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:30:11.821 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:30:11.821 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:30:11.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:30:11.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:30:11.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:30:11.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:30:11.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4f0f03ab-2bed-4777-a38b-0fb0e2cbdd38 2026-02-22T20:30:11.823 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 4f0f03ab-2bed-4777-a38b-0fb0e2cbdd38 2026-02-22T20:30:11.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4f0f03ab-2bed-4777-a38b-0fb0e2cbdd38' 2026-02-22T20:30:11.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:30:11.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBTZ5tpivJdMRAAVSNzKQslTctHvdv5rvDWzA== 2026-02-22T20:30:11.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBTZ5tpivJdMRAAVSNzKQslTctHvdv5rvDWzA=="}' 2026-02-22T20:30:11.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4f0f03ab-2bed-4777-a38b-0fb0e2cbdd38 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:30:11.925 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:30:11.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:30:11.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTZ5tpivJdMRAAVSNzKQslTctHvdv5rvDWzA== --osd-uuid 4f0f03ab-2bed-4777-a38b-0fb0e2cbdd38 2026-02-22T20:30:11.950 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:11.943+0000 7fbc4c628a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:11.952 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:11.947+0000 7fbc4c628a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:11.953 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:11.947+0000 7fbc4c628a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:11.954 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:11.947+0000 7fbc4c628a40 -1 bdev(0x56164e570c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:30:11.954 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:11.947+0000 7fbc4c628a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:30:12.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:30:12.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:30:12.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:30:12.696 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:30:12.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:30:12.893 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:30:12.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:30:12.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:30:12.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:30:12.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:30:12.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:30:12.917 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:12.911+0000 7f488c1dda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:12.919 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:12.911+0000 7f488c1dda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:12.920 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:12.915+0000 7f488c1dda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:13.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:30:13.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:30:13.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:30:13.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:30:13.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:30:13.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:13.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:30:13.056 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:30:13.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:13.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:30:13.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:13.342 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:13.335+0000 7f488c1dda40 -1 Falling back to public interface 2026-02-22T20:30:13.931 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:13.923+0000 7f488c1dda40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:30:14.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:14.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:14.219 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:30:14.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:30:14.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:14.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:30:14.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:15.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:15.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:15.388 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:30:15.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:30:15.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:15.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:30:15.546 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2498682675,v1:127.0.0.1:6803/2498682675] [v2:127.0.0.1:6804/2498682675,v1:127.0.0.1:6805/2498682675] exists,up 4f0f03ab-2bed-4777-a38b-0fb0e2cbdd38 2026-02-22T20:30:15.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:30:15.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:30:15.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:30:15.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-22T20:30:15.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 1 2026-02-22T20:30:15.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:30:15.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:30:15.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:30:15.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:30:15.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:30:15.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:30:15.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:30:15.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:30:15.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:30:15.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:30:15.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:30:15.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:30:15.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:30:15.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:30:15.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:30:15.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:30:15.552 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:30:15.552 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:30:15.552 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:15.552 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:15.552 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:15.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:15.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:30:15.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:30:15.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:30:15.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:30:15.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:30:15.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:30:15.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:30:15.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:30:15.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:30:15.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:30:15.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:30:15.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:30:15.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:30:15.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=03c0b956-e207-4fd8-aad4-3d09bde31c58 2026-02-22T20:30:15.554 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 03c0b956-e207-4fd8-aad4-3d09bde31c58 2026-02-22T20:30:15.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 03c0b956-e207-4fd8-aad4-3d09bde31c58' 2026-02-22T20:30:15.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:30:15.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBXZ5tpCtBLIRAAZ533BW3SheXR/22B4jKPzg== 2026-02-22T20:30:15.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBXZ5tpCtBLIRAAZ533BW3SheXR/22B4jKPzg=="}' 2026-02-22T20:30:15.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 03c0b956-e207-4fd8-aad4-3d09bde31c58 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:30:15.728 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:30:15.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:30:15.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXZ5tpCtBLIRAAZ533BW3SheXR/22B4jKPzg== --osd-uuid 03c0b956-e207-4fd8-aad4-3d09bde31c58 2026-02-22T20:30:15.755 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:15.747+0000 7f2c8c4cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:15.756 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:15.751+0000 7f2c8c4cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:15.757 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:15.751+0000 7f2c8c4cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:15.758 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:15.751+0000 7f2c8c4cfa40 -1 bdev(0x55f0352b6c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:30:15.758 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:15.751+0000 7f2c8c4cfa40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:30:16.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:30:16.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:30:16.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:30:16.449 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:30:16.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:30:16.643 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:30:16.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:30:16.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:30:16.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:30:16.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:30:16.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:30:16.667 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:16.659+0000 7fdc04c9ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:16.669 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:16.663+0000 7fdc04c9ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:16.670 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:16.663+0000 7fdc04c9ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:16.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:30:16.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:30:16.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:30:16.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:30:16.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:30:16.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:16.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:30:16.803 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:30:16.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:16.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:30:16.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:16.998 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:16.991+0000 7fdc04c9ba40 -1 Falling back to public interface 2026-02-22T20:30:17.566 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:17.559+0000 7fdc04c9ba40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:30:17.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:17.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:17.960 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:30:17.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:30:17.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:17.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:30:18.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:19.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:19.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:19.126 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:30:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:30:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:30:19.283 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3849295862,v1:127.0.0.1:6811/3849295862] [v2:127.0.0.1:6812/3849295862,v1:127.0.0.1:6813/3849295862] exists,up 03c0b956-e207-4fd8-aad4-3d09bde31c58 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 2 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:30:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:30:19.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:30:19.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:30:19.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:30:19.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:30:19.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:30:19.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:30:19.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:30:19.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:30:19.285 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:30:19.285 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:19.285 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:19.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:30:19.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:30:19.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:30:19.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fc6e0473-9175-4301-9e11-fc7b02a1d330 2026-02-22T20:30:19.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 fc6e0473-9175-4301-9e11-fc7b02a1d330' 2026-02-22T20:30:19.289 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 fc6e0473-9175-4301-9e11-fc7b02a1d330 2026-02-22T20:30:19.289 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:30:19.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBbZ5tp4RWrERAAoW5KzM7s2rT5ZJMzgLKGZA== 2026-02-22T20:30:19.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBbZ5tp4RWrERAAoW5KzM7s2rT5ZJMzgLKGZA=="}' 2026-02-22T20:30:19.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fc6e0473-9175-4301-9e11-fc7b02a1d330 -i td/osd-scrub-repair/2/new.json 2026-02-22T20:30:19.470 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:30:19.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:30:19.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBbZ5tp4RWrERAAoW5KzM7s2rT5ZJMzgLKGZA== --osd-uuid fc6e0473-9175-4301-9e11-fc7b02a1d330 2026-02-22T20:30:19.497 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:19.491+0000 7f8c5be50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:19.499 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:19.491+0000 7f8c5be50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:19.500 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:19.491+0000 7f8c5be50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:19.500 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:19.495+0000 7f8c5be50a40 -1 bdev(0x55986032ac00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:30:19.500 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:19.495+0000 7f8c5be50a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:30:20.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:30:20.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:30:20.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:30:20.120 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:30:20.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:30:20.324 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:30:20.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:30:20.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:30:20.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:30:20.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:30:20.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:30:20.349 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:20.343+0000 7fb4a1294a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:20.351 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:20.343+0000 7fb4a1294a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:20.352 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:20.347+0000 7fb4a1294a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:20.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:30:20.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:30:20.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:30:20.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:30:20.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:30:20.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:20.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:30:20.483 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:30:20.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:20.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:30:20.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:20.770 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:20.763+0000 7fb4a1294a40 -1 Falling back to public interface 2026-02-22T20:30:21.302 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:21.295+0000 7fb4a1294a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:30:21.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:21.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:21.648 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:30:21.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:30:21.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:21.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:30:21.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:22.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:22.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:22.811 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:30:22.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:30:22.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:22.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:30:22.972 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1886322852,v1:127.0.0.1:6819/1886322852] [v2:127.0.0.1:6820/1886322852,v1:127.0.0.1:6821/1886322852] exists,up fc6e0473-9175-4301-9e11-fc7b02a1d330 2026-02-22T20:30:22.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:30:22.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:30:22.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:30:22.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:863: corrupt_and_repair_jerasure: for id in $(seq 0 3) 2026-02-22T20:30:22.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:864: corrupt_and_repair_jerasure: run_osd td/osd-scrub-repair 3 2026-02-22T20:30:22.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:30:22.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:22.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:30:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=68dadeef-d802-40cc-aa88-89d20bdaf7d9 2026-02-22T20:30:22.977 INFO:tasks.workunit.client.0.trial150.stdout:add osd3 68dadeef-d802-40cc-aa88-89d20bdaf7d9 2026-02-22T20:30:22.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 68dadeef-d802-40cc-aa88-89d20bdaf7d9' 2026-02-22T20:30:22.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:30:22.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBeZ5tpc0WqOhAAWpI2C5ouIZ61QOt+tchlKg== 2026-02-22T20:30:22.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBeZ5tpc0WqOhAAWpI2C5ouIZ61QOt+tchlKg=="}' 2026-02-22T20:30:22.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 68dadeef-d802-40cc-aa88-89d20bdaf7d9 -i td/osd-scrub-repair/3/new.json 2026-02-22T20:30:23.148 INFO:tasks.workunit.client.0.trial150.stdout:3 2026-02-22T20:30:23.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-22T20:30:23.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBeZ5tpc0WqOhAAWpI2C5ouIZ61QOt+tchlKg== --osd-uuid 68dadeef-d802-40cc-aa88-89d20bdaf7d9 2026-02-22T20:30:23.174 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:23.167+0000 7f4cd48dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:23.176 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:23.167+0000 7f4cd48dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:23.177 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:23.171+0000 7f4cd48dfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:23.177 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:23.171+0000 7f4cd48dfa40 -1 bdev(0x562092e6ac00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-22T20:30:23.177 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:23.171+0000 7f4cd48dfa40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-22T20:30:23.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-22T20:30:23.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:30:23.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-22T20:30:23.809 INFO:tasks.workunit.client.0.trial150.stdout:adding osd3 key to auth repository 2026-02-22T20:30:23.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:30:24.011 INFO:tasks.workunit.client.0.trial150.stdout:start osd.3 2026-02-22T20:30:24.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-22T20:30:24.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:30:24.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:30:24.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:30:24.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:30:24.029 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:24.023+0000 7f09e4771a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:24.030 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:24.023+0000 7f09e4771a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:24.031 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:24.023+0000 7f09e4771a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:24.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-22T20:30:24.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:30:24.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:30:24.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:30:24.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:30:24.171 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:30:24.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:24.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:30:24.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:24.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:30:24.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:24.382 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:24.375+0000 7f09e4771a40 -1 Falling back to public interface 2026-02-22T20:30:25.030 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:25.023+0000 7f09e4771a40 -1 osd.3 0 log_to_monitors true 2026-02-22T20:30:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:25.334 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:30:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:30:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:30:25.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:26.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:26.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:26.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:30:26.501 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:30:26.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:26.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:30:26.659 INFO:tasks.workunit.client.0.trial150.stdout:osd.3 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4027689617,v1:127.0.0.1:6827/4027689617] [v2:127.0.0.1:6828/4027689617,v1:127.0.0.1:6829/4027689617] exists,up 68dadeef-d802-40cc-aa88-89d20bdaf7d9 2026-02-22T20:30:26.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:30:26.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:30:26.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:30:26.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:866: corrupt_and_repair_jerasure: create_rbd_pool 2026-02-22T20:30:26.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:30:26.805 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:30:26.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:30:26.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:30:27.030 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:30:27.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:30:28.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:30:28.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:867: corrupt_and_repair_jerasure: wait_for_clean 2026-02-22T20:30:28.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:30:28.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:30:28.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:30:28.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:30:28.323 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:30:28.323 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:30:28.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:30:28.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:30:28.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:30:28.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:30:28.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:30:28.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:30:28.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:30:28.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:30:28.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:30:28.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:30:28.587 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:30:28.587 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:30:28.587 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:30:28.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:30:28.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:28.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:30:28.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-22T20:30:28.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-22T20:30:28.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T20:30:28.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:28.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:30:28.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-22T20:30:28.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-22T20:30:28.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705668' 2026-02-22T20:30:28.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:28.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:30:28.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574851 2026-02-22T20:30:28.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574851 2026-02-22T20:30:28.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705668 2-55834574851' 2026-02-22T20:30:28.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:28.808 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:30:28.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444034 2026-02-22T20:30:28.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444034 2026-02-22T20:30:28.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705668 2-55834574851 3-73014444034' 2026-02-22T20:30:28.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:28.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-22T20:30:28.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:28.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:30:28.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-22T20:30:28.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:28.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-22T20:30:28.883 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836484 2026-02-22T20:30:28.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T20:30:28.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:29.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:30:29.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:30:30.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:30:30.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:30.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:30:30.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:30:31.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:30:31.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:31.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836484 2026-02-22T20:30:31.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:31.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-22T20:30:31.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:31.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:30:31.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705668 2026-02-22T20:30:31.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:31.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705668 2026-02-22T20:30:31.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-22T20:30:31.362 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705668 2026-02-22T20:30:31.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:30:31.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705668 2026-02-22T20:30:31.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:31.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574851 2026-02-22T20:30:31.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:31.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:30:31.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574851 2026-02-22T20:30:31.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:31.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574851 2026-02-22T20:30:31.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574851' 2026-02-22T20:30:31.528 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574851 2026-02-22T20:30:31.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:30:31.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574851 -lt 55834574851 2026-02-22T20:30:31.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:31.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-73014444034 2026-02-22T20:30:31.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:31.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:30:31.703 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-73014444034 2026-02-22T20:30:31.704 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:31.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=73014444034 2026-02-22T20:30:31.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 73014444034' 2026-02-22T20:30:31.705 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.3 seq 73014444034 2026-02-22T20:30:31.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:30:31.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 73014444034 -lt 73014444034 2026-02-22T20:30:31.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:30:31.865 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:31.865 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:32.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:30:32.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:30:32.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:30:32.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:30:32.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:30:32.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:30:32.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:30:32.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:30:32.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:30:32.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:30:32.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:32.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:32.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:30:32.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:30:32.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:30:32.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:869: corrupt_and_repair_jerasure: create_ec_pool ecpool true k=2 m=2 2026-02-22T20:30:32.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-22T20:30:32.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:30:32.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-22T20:30:32.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:30:32.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2026-02-22T20:30:32.605 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:30:32.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-22T20:30:32.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-22T20:30:32.910 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' created 2026-02-22T20:30:32.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:30:33.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-22T20:30:33.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-22T20:30:34.127 INFO:tasks.workunit.client.0.trial150.stderr:set pool 2 allow_ec_overwrites to true 2026-02-22T20:30:34.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:30:34.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:30:34.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:30:34.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:30:34.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:30:34.139 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:30:34.139 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:30:34.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:30:34.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:30:34.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:30:34.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:30:34.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:30:34.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:30:34.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:30:34.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:30:34.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:30:34.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:30:34.409 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:30:34.410 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:30:34.410 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:30:34.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:30:34.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:34.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:30:34.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-22T20:30:34.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-22T20:30:34.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-22T20:30:34.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:34.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:30:34.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705670 2026-02-22T20:30:34.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705670 2026-02-22T20:30:34.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705670' 2026-02-22T20:30:34.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:34.562 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:30:34.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574853 2026-02-22T20:30:34.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574853 2026-02-22T20:30:34.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705670 2-55834574853' 2026-02-22T20:30:34.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:34.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:30:34.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=73014444036 2026-02-22T20:30:34.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 73014444036 2026-02-22T20:30:34.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705670 2-55834574853 3-73014444036' 2026-02-22T20:30:34.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:34.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-22T20:30:34.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:34.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:30:34.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-22T20:30:34.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:34.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-22T20:30:34.711 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836486 2026-02-22T20:30:34.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-22T20:30:34.712 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:34.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-22T20:30:34.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:30:35.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:30:35.882 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:36.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-22T20:30:36.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:30:37.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:30:37.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:37.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836486 2026-02-22T20:30:37.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:37.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705670 2026-02-22T20:30:37.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:37.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:30:37.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705670 2026-02-22T20:30:37.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:37.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705670 2026-02-22T20:30:37.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2026-02-22T20:30:37.207 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705670 2026-02-22T20:30:37.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:30:37.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705670 -lt 38654705670 2026-02-22T20:30:37.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:37.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574853 2026-02-22T20:30:37.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:37.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:30:37.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574853 2026-02-22T20:30:37.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:37.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574853 2026-02-22T20:30:37.376 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574853 2026-02-22T20:30:37.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574853' 2026-02-22T20:30:37.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:30:37.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574853 -lt 55834574853 2026-02-22T20:30:37.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:37.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-73014444036 2026-02-22T20:30:37.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:37.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:30:37.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-73014444036 2026-02-22T20:30:37.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:37.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=73014444036 2026-02-22T20:30:37.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 73014444036' 2026-02-22T20:30:37.543 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.3 seq 73014444036 2026-02-22T20:30:37.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:30:37.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 73014444037 -lt 73014444036 2026-02-22T20:30:37.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:30:37.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:37.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:37.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:30:37.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:30:37.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:30:37.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:30:37.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:30:37.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:30:37.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:30:37.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:30:38.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:30:38.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:30:38.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:38.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:38.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:30:38.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:30:38.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:30:38.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:30:38.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:870: corrupt_and_repair_jerasure: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2026-02-22T20:30:38.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-22T20:30:38.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: corrupt_and_repair_erasure_coded: local poolname=ecpool 2026-02-22T20:30:38.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-22T20:30:38.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:30:38.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:30:38.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-22T20:30:38.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:30:38.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:30:38.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:30:38.433 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:30:38.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:30:38.634 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:30:38.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:30:38.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:30:38.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:30:38.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2026-02-22T20:30:38.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-22T20:30:38.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:30:38.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:30:38.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:30:38.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: local primary=3 2026-02-22T20:30:38.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2026-02-22T20:30:38.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-22T20:30:38.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: sed -e s/3// 2026-02-22T20:30:38.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-22T20:30:38.825 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:30:38.825 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:30:38.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-22T20:30:38.982 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:30:38.982 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:30:38.982 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:30:38.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 1 2 0 2026-02-22T20:30:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: osds=('1' '2' '0') 2026-02-22T20:30:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: local -a osds 2026-02-22T20:30:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: corrupt_and_repair_erasure_coded: local not_primary_first=1 2026-02-22T20:30:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: corrupt_and_repair_erasure_coded: local not_primary_second=2 2026-02-22T20:30:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2026-02-22T20:30:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-22T20:30:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-22T20:30:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=3 2026-02-22T20:30:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:30:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:30:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:30:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:30:38.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:30:38.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:30:38.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:30:38.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:30:38.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:30:38.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:30:38.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:30:38.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:30:38.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:30:38.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:30:38.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:30:38.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:30:39.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:30:39.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:30:39.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:30:39.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:30:39.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:30:39.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:30:39.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:30:39.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-22T20:30:39.488 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:30:39.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:30:39.578 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:30:39.578 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:39.578 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:39.578 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:39.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:39.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:30:39.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:30:39.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:30:39.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:30:39.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:30:39.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:30:39.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:30:39.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:30:39.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:30:39.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:30:39.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:30:39.579 INFO:tasks.workunit.client.0.trial150.stderr:start osd.3 2026-02-22T20:30:39.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:30:39.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:30:39.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:30:39.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:30:39.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:30:39.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:30:39.598 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:39.591+0000 7ff4bbec9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:39.599 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:39.591+0000 7ff4bbec9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:39.600 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:39.591+0000 7ff4bbec9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:39.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:30:39.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:30:39.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:30:39.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:30:39.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:30:39.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:39.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:30:39.740 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:30:39.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:39.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:30:39.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:39.982 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:39.975+0000 7ff4bbec9a40 -1 Falling back to public interface 2026-02-22T20:30:40.608 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:40.599+0000 7ff4bbec9a40 -1 osd.3 33 log_to_monitors true 2026-02-22T20:30:40.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:40.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:40.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:30:40.906 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:30:40.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:40.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:30:41.062 INFO:tasks.workunit.client.0.trial150.stderr:osd.3 up in weight 1 up_from 37 up_thru 37 down_at 34 last_clean_interval [17,33) [v2:127.0.0.1:6826/2975313665,v1:127.0.0.1:6827/2975313665] [v2:127.0.0.1:6828/2975313665,v1:127.0.0.1:6829/2975313665] exists,up 68dadeef-d802-40cc-aa88-89d20bdaf7d9 2026-02-22T20:30:41.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:30:41.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:30:41.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:30:41.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:30:41.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:30:41.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:30:41.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:30:41.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:30:41.063 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:30:41.063 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:30:41.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:30:41.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:30:41.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:30:41.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:30:41.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:30:41.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:30:41.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:30:41.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:30:41.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:30:41.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:30:41.333 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:30:41.333 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:30:41.333 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:30:41.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:30:41.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:41.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:30:41.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-22T20:30:41.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-22T20:30:41.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-22T20:30:41.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:41.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:30:41.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705672 2026-02-22T20:30:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705672 2026-02-22T20:30:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672' 2026-02-22T20:30:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:41.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:30:41.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574855 2026-02-22T20:30:41.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574855 2026-02-22T20:30:41.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855' 2026-02-22T20:30:41.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:41.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:30:41.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789954 2026-02-22T20:30:41.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789954 2026-02-22T20:30:41.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574855 3-158913789954' 2026-02-22T20:30:41.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:41.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-22T20:30:41.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:41.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:30:41.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-22T20:30:41.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:41.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-22T20:30:41.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-22T20:30:41.629 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836489 2026-02-22T20:30:41.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:41.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836489 2026-02-22T20:30:41.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:30:42.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:30:42.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:42.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-22T20:30:42.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:42.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705672 2026-02-22T20:30:42.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:42.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:30:42.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705672 2026-02-22T20:30:42.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:42.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705672 2026-02-22T20:30:42.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-22T20:30:42.956 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 38654705672 2026-02-22T20:30:42.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:30:43.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-22T20:30:43.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:43.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574855 2026-02-22T20:30:43.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:43.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:30:43.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574855 2026-02-22T20:30:43.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:43.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574855 2026-02-22T20:30:43.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574855' 2026-02-22T20:30:43.134 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 55834574855 2026-02-22T20:30:43.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:30:43.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574855 -lt 55834574855 2026-02-22T20:30:43.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:43.289 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-158913789954 2026-02-22T20:30:43.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:43.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:30:43.292 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-158913789954 2026-02-22T20:30:43.292 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:43.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789954 2026-02-22T20:30:43.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 158913789954' 2026-02-22T20:30:43.293 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 158913789954 2026-02-22T20:30:43.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:30:43.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789954 -lt 158913789954 2026-02-22T20:30:43.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:30:43.455 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:43.455 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:43.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:30:43.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:30:43.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:30:43.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:30:43.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:30:43.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:30:43.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:30:43.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:30:43.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:30:43.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:30:43.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:43.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:43.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:30:43.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:30:43.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:30:43.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-22T20:30:43.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:30:43.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:30:43.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:30:43.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:30:44.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-22T20:30:44.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-22T20:30:44.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:30:44.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:30:44.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:30:44.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:30:44.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:30:44.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:30:44.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:30:32.904841+0000 2026-02-22T20:30:44.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:30:44.464 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:30:44.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:30:32.904841+0000 2026-02-22T20:30:44.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:30:44.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:30:32.904841+0000 2026-02-22T20:30:44.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:30:44.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:30:44.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:30:44.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:30:44.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:30:44.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:30:44.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:30:44.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:30:44.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:32.904841+0000 '>' 2026-02-22T20:30:32.904841+0000 2026-02-22T20:30:44.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:30:45.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:30:45.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:30:45.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:30:45.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:30:45.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:30:45.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:30:45.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:30:45.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:32.904841+0000 '>' 2026-02-22T20:30:32.904841+0000 2026-02-22T20:30:45.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:30:46.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:30:46.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:30:46.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:30:46.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:30:46.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:30:46.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:30:46.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:44.695907+0000 '>' 2026-02-22T20:30:32.904841+0000 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:30:46.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:30:47.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:30:47.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:30:47.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:30:47.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:30:47.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:30:47.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:30:47.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:30:47.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-22T20:30:47.339 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-22T20:30:47.339 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-22T20:30:47.340 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:30:47.340 INFO:tasks.workunit.client.0.trial150.stdout:hinfo_key 2026-02-22T20:30:47.340 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:30:47.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:30:47.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:30:47.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:30:47.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:30:47.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:30:47.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:30:47.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:30:47.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:30:47.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:30:47.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:30:47.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:30:47.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:30:47.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:30:47.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:30:47.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:30:47.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:30:47.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:30:47.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:30:47.425 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:30:47.426 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:47.426 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:47.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:47.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:47.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:30:47.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:30:47.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:30:47.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:30:47.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:30:47.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:30:47.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:30:47.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:30:47.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:30:47.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:30:47.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:30:47.428 INFO:tasks.workunit.client.0.trial150.stderr:start osd.3 2026-02-22T20:30:47.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:30:47.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:30:47.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:30:47.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:30:47.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:30:47.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:30:47.444 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:47.434+0000 7fad96e79a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:47.445 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:47.438+0000 7fad96e79a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:47.446 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:47.438+0000 7fad96e79a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:47.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:30:47.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:30:47.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:30:47.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:30:47.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:30:47.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:47.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:30:47.585 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:30:47.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:47.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:30:47.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:47.810 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:47.802+0000 7fad96e79a40 -1 Falling back to public interface 2026-02-22T20:30:48.444 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:48.434+0000 7fad96e79a40 -1 osd.3 39 log_to_monitors true 2026-02-22T20:30:48.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:48.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:48.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:30:48.748 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:30:48.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:48.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:30:48.900 INFO:tasks.workunit.client.0.trial150.stderr:osd.3 up in weight 1 up_from 43 up_thru 43 down_at 40 last_clean_interval [37,39) [v2:127.0.0.1:6826/1995762926,v1:127.0.0.1:6827/1995762926] [v2:127.0.0.1:6828/1995762926,v1:127.0.0.1:6829/1995762926] exists,up 68dadeef-d802-40cc-aa88-89d20bdaf7d9 2026-02-22T20:30:48.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:30:48.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:30:48.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:30:48.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:30:48.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:30:48.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:30:48.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:30:48.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:30:48.902 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:30:48.902 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:30:48.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:30:48.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:30:48.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:30:49.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:30:49.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:30:49.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:30:49.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:30:49.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:30:49.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:30:49.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:30:49.169 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:30:49.169 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:30:49.169 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:30:49.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:30:49.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:49.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:30:49.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-22T20:30:49.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-22T20:30:49.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T20:30:49.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:49.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:30:49.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705675 2026-02-22T20:30:49.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705675 2026-02-22T20:30:49.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705675' 2026-02-22T20:30:49.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:49.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:30:49.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574858 2026-02-22T20:30:49.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574858 2026-02-22T20:30:49.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705675 2-55834574858' 2026-02-22T20:30:49.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:49.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:30:49.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593730 2026-02-22T20:30:49.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593730 2026-02-22T20:30:49.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705675 2-55834574858 3-184683593730' 2026-02-22T20:30:49.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:49.466 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-22T20:30:49.466 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:49.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:30:49.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-22T20:30:49.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:49.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-22T20:30:49.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T20:30:49.470 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836491 2026-02-22T20:30:49.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:49.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T20:30:49.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:30:50.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:30:50.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:50.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836491 2026-02-22T20:30:50.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:50.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705675 2026-02-22T20:30:50.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:50.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:30:50.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705675 2026-02-22T20:30:50.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:50.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705675 2026-02-22T20:30:50.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705675' 2026-02-22T20:30:50.800 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 38654705675 2026-02-22T20:30:50.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:30:50.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705675 -lt 38654705675 2026-02-22T20:30:50.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:50.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574858 2026-02-22T20:30:50.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:50.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:30:50.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574858 2026-02-22T20:30:50.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:50.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574858 2026-02-22T20:30:50.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574858' 2026-02-22T20:30:50.975 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 55834574858 2026-02-22T20:30:50.976 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:30:51.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574858 -lt 55834574858 2026-02-22T20:30:51.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:51.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-184683593730 2026-02-22T20:30:51.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:51.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:30:51.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-184683593730 2026-02-22T20:30:51.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:51.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593730 2026-02-22T20:30:51.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 184683593730' 2026-02-22T20:30:51.137 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 184683593730 2026-02-22T20:30:51.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:30:51.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593730 -lt 184683593730 2026-02-22T20:30:51.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:30:51.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:51.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:51.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:30:51.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:30:51.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:30:51.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:30:51.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:30:51.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:30:51.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:30:51.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:30:51.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:30:51.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:30:51.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:51.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:51.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:30:51.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:30:51.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:30:51.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:30:51.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:30:51.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:321: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 1 2026-02-22T20:30:51.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-22T20:30:51.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-22T20:30:51.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=1 2026-02-22T20:30:51.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:30:51.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:30:51.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:30:51.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:30:51.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:30:51.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:30:51.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:30:51.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:30:51.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:30:51.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:30:51.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:30:51.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:30:51.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:30:51.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:30:51.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:30:51.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:30:52.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:30:52.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:30:52.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:30:52.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:30:52.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:30:52.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:30:52.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:30:52.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-22T20:30:52.377 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:30:52.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:30:52.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:30:52.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:30:52.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:30:52.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:30:52.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:30:52.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:30:52.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:30:52.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:30:52.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:30:52.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:30:52.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:30:52.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:30:52.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:30:52.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:30:52.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:30:52.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:30:52.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:30:52.478 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:30:52.478 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:30:52.478 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:30:52.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:52.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:30:52.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:30:52.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:30:52.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:30:52.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:30:52.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:30:52.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:30:52.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:30:52.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:30:52.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:30:52.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:30:52.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:30:52.480 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:30:52.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:30:52.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:30:52.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:30:52.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:30:52.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:30:52.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:30:52.500 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:52.490+0000 7f136cad8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:52.501 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:52.494+0000 7f136cad8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:52.502 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:52.494+0000 7f136cad8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:30:52.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:30:52.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:30:52.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:30:52.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:30:52.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:30:52.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:52.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:30:52.645 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:30:52.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:52.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:30:52.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:30:52.882 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:52.874+0000 7f136cad8a40 -1 Falling back to public interface 2026-02-22T20:30:53.460 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:30:53.450+0000 7f136cad8a40 -1 osd.1 45 log_to_monitors true 2026-02-22T20:30:53.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:30:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:30:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:30:53.804 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:30:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:30:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:30:53.962 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 49 up_thru 49 down_at 46 last_clean_interval [9,45) [v2:127.0.0.1:6810/3713398335,v1:127.0.0.1:6811/3713398335] [v2:127.0.0.1:6812/3713398335,v1:127.0.0.1:6813/3713398335] exists,up 03c0b956-e207-4fd8-aad4-3d09bde31c58 2026-02-22T20:30:53.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:30:53.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:30:53.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:30:53.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:30:53.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:30:53.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:30:53.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:30:53.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:30:53.964 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:30:53.964 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:30:53.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:30:53.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:30:53.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:30:54.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:30:54.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:30:54.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:30:54.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:30:54.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:30:54.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:30:54.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:30:54.237 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:30:54.237 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:30:54.237 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:30:54.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:30:54.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:54.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:30:54.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-22T20:30:54.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-22T20:30:54.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-22T20:30:54.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:54.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:30:54.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397506 2026-02-22T20:30:54.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397506 2026-02-22T20:30:54.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-210453397506' 2026-02-22T20:30:54.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:54.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:30:54.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574860 2026-02-22T20:30:54.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574860 2026-02-22T20:30:54.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-210453397506 2-55834574860' 2026-02-22T20:30:54.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:30:54.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:30:54.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593732 2026-02-22T20:30:54.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593732 2026-02-22T20:30:54.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-210453397506 2-55834574860 3-184683593732' 2026-02-22T20:30:54.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:54.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-22T20:30:54.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:54.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:30:54.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-22T20:30:54.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:54.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-22T20:30:54.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-22T20:30:54.533 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836493 2026-02-22T20:30:54.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:54.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-22T20:30:54.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:30:55.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:30:55.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:55.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-22T20:30:55.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:30:56.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:30:56.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:30:57.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836493 2026-02-22T20:30:57.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:57.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-210453397506 2026-02-22T20:30:57.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:57.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:30:57.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-210453397506 2026-02-22T20:30:57.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:57.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=210453397506 2026-02-22T20:30:57.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 210453397506' 2026-02-22T20:30:57.018 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 210453397506 2026-02-22T20:30:57.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:30:57.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 210453397506 -lt 210453397506 2026-02-22T20:30:57.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:57.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574860 2026-02-22T20:30:57.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:57.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:30:57.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574860 2026-02-22T20:30:57.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:57.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574860 2026-02-22T20:30:57.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574860' 2026-02-22T20:30:57.191 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 55834574860 2026-02-22T20:30:57.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:30:57.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574860 -lt 55834574860 2026-02-22T20:30:57.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:30:57.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-184683593732 2026-02-22T20:30:57.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:30:57.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:30:57.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-184683593732 2026-02-22T20:30:57.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:30:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593732 2026-02-22T20:30:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 184683593732' 2026-02-22T20:30:57.361 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 184683593732 2026-02-22T20:30:57.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:30:57.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593732 -lt 184683593732 2026-02-22T20:30:57.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:30:57.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:57.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:57.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:30:57.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:30:57.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:30:57.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:30:57.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:30:57.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:30:57.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:30:57.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:30:57.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:30:57.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:30:57.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:30:57.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:30:58.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:30:58.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:30:58.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:30:58.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-22T20:30:58.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:30:58.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:30:58.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:30:58.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:30:58.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-22T20:30:58.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-22T20:30:58.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:30:58.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:30:58.233 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:30:58.233 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:30:58.233 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:30:58.233 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:30:58.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:30:44.695907+0000 2026-02-22T20:30:58.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:30:58.539 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:30:58.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:30:44.695907+0000 2026-02-22T20:30:58.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:30:58.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:30:44.695907+0000 2026-02-22T20:30:58.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:30:58.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:30:58.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:30:58.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:30:58.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:30:58.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:30:58.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:30:58.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:30:58.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:44.695907+0000 '>' 2026-02-22T20:30:44.695907+0000 2026-02-22T20:30:58.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:30:59.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:30:59.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:30:59.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:30:59.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:30:59.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:30:59.711 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:30:59.711 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:30:59.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:44.695907+0000 '>' 2026-02-22T20:30:44.695907+0000 2026-02-22T20:30:59.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:00.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:00.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:00.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:00.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:00.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:00.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:00.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:01.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:44.695907+0000 '>' 2026-02-22T20:30:44.695907+0000 2026-02-22T20:31:01.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:02.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:02.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:02.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:02.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:02.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:02.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:02.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:02.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:44.695907+0000 '>' 2026-02-22T20:30:44.695907+0000 2026-02-22T20:31:02.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:03.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:03.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:03.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:03.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:03.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:03.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:03.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:03.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:44.695907+0000 '>' 2026-02-22T20:30:44.695907+0000 2026-02-22T20:31:03.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:04.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:04.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:04.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:04.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:04.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:04.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:04.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:04.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:59.399491+0000 '>' 2026-02-22T20:30:44.695907+0000 2026-02-22T20:31:04.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:31:04.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:31:04.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:31:04.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:31:04.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:31:04.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:31:04.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:31:04.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:31:04.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:31:04.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:31:04.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:31:04.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:31:04.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:31:04.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:31:04.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:31:04.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:31:04.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:31:04.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:31:04.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:31:04.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:31:04.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:31:04.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:31:04.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:31:04.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:31:04.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-22T20:31:04.880 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:31:04.880 INFO:tasks.workunit.client.0.trial150.stdout:hinfo_key 2026-02-22T20:31:04.880 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:31:04.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:31:04.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:04.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:31:04.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:31:04.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:31:04.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:04.979 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:04.980 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:04.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:04.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:04.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:04.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:04.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:04.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:04.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:04.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:04.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:04.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:31:04.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:31:04.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:31:04.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:31:04.982 INFO:tasks.workunit.client.0.trial150.stderr:start osd.3 2026-02-22T20:31:04.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:31:04.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:31:04.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:31:04.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:31:04.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:31:04.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:31:05.000 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:04.990+0000 7f5fa022fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:05.001 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:04.994+0000 7f5fa022fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:05.002 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:04.994+0000 7f5fa022fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:05.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:31:05.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:05.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:31:05.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:05.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:05.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:05.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:05.147 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:31:05.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:05.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:31:05.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:05.366 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:05.358+0000 7f5fa022fa40 -1 Falling back to public interface 2026-02-22T20:31:05.984 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:05.974+0000 7f5fa022fa40 -1 osd.3 52 log_to_monitors true 2026-02-22T20:31:06.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:06.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:06.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:06.307 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:31:06.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:06.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:31:06.463 INFO:tasks.workunit.client.0.trial150.stderr:osd.3 up in weight 1 up_from 55 up_thru 55 down_at 52 last_clean_interval [43,52) [v2:127.0.0.1:6826/4192352170,v1:127.0.0.1:6827/4192352170] [v2:127.0.0.1:6828/4192352170,v1:127.0.0.1:6829/4192352170] exists,up 68dadeef-d802-40cc-aa88-89d20bdaf7d9 2026-02-22T20:31:06.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:31:06.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:31:06.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:31:06.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:31:06.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:31:06.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:31:06.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:31:06.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:31:06.464 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:31:06.464 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:31:06.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:31:06.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:31:06.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:31:06.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:31:06.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:31:06.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:31:06.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:31:06.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:31:06.578 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:31:06.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:31:06.741 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:31:06.741 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:31:06.741 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:31:06.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:31:06.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:06.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:31:06.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-22T20:31:06.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-22T20:31:06.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-22T20:31:06.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:06.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:31:06.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397509 2026-02-22T20:31:06.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397509 2026-02-22T20:31:06.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-210453397509' 2026-02-22T20:31:06.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:06.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:31:06.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574863 2026-02-22T20:31:06.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574863 2026-02-22T20:31:06.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-210453397509 2-55834574863' 2026-02-22T20:31:06.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:06.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:31:07.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=236223201282 2026-02-22T20:31:07.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 236223201282 2026-02-22T20:31:07.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-210453397509 2-55834574863 3-236223201282' 2026-02-22T20:31:07.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:07.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-22T20:31:07.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:07.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:31:07.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-22T20:31:07.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:07.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-22T20:31:07.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-22T20:31:07.039 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836497 2026-02-22T20:31:07.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:07.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836497 2026-02-22T20:31:07.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:31:08.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:31:08.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:08.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836497 2026-02-22T20:31:08.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:08.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-210453397509 2026-02-22T20:31:08.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:08.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:31:08.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-210453397509 2026-02-22T20:31:08.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:08.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=210453397509 2026-02-22T20:31:08.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 210453397509' 2026-02-22T20:31:08.371 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 210453397509 2026-02-22T20:31:08.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:31:08.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 210453397509 -lt 210453397509 2026-02-22T20:31:08.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:08.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574863 2026-02-22T20:31:08.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:08.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:31:08.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574863 2026-02-22T20:31:08.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:08.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574863 2026-02-22T20:31:08.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574863' 2026-02-22T20:31:08.543 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 55834574863 2026-02-22T20:31:08.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:08.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574864 -lt 55834574863 2026-02-22T20:31:08.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:08.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-236223201282 2026-02-22T20:31:08.714 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:08.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:31:08.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-236223201282 2026-02-22T20:31:08.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:08.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=236223201282 2026-02-22T20:31:08.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 236223201282' 2026-02-22T20:31:08.717 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 236223201282 2026-02-22T20:31:08.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:31:08.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 236223201282 -lt 236223201282 2026-02-22T20:31:08.882 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:31:08.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:08.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:09.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:31:09.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:09.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:09.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:09.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:09.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:09.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:09.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:09.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:31:09.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:09.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:09.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:09.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:31:09.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:31:09.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:31:09.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:31:09.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:31:09.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 1 2 2026-02-22T20:31:09.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-22T20:31:09.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-22T20:31:09.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=1 2026-02-22T20:31:09.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=2 2026-02-22T20:31:09.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-22T20:31:09.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:31:09.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:31:09.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:31:09.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 155843"' 2026-02-22T20:31:09.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 155843' 2026-02-22T20:31:09.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:31:09.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:31:09.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:31:09.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 155845"' 2026-02-22T20:31:09.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:31:09.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 155845' 2026-02-22T20:31:09.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-22T20:31:09.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 155843 155845' 2026-02-22T20:31:09.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:31:09.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:31:09.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 155843 2026-02-22T20:31:09.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/155846: /' 2026-02-22T20:31:09.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:31:09.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/155848: /' 2026-02-22T20:31:10.065 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:31:10.065 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:10.066 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:31:10.069 INFO:tasks.workunit.client.0.trial150.stderr:155846: start osd.1 2026-02-22T20:31:10.070 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:31:10.089 INFO:tasks.workunit.client.0.trial150.stderr:1155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: remove 2#2:eb822e21:::SOMETHING:head# 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-22T20:31:10.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:10.091 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:31:10.091 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:31:10.091 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:31:10.091 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:31:10.091 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:10.091 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:10.093 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: start osd.2 2026-02-22T20:31:10.094 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:31:11.548 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:31:11.548 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: 2026-02-22T20:31:10.078+0000 7f3a1949ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: 2026-02-22T20:31:10.078+0000 7f3a1949ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: 2026-02-22T20:31:10.082+0000 7f3a1949ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: 0 2026-02-22T20:31:11.549 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: 2026-02-22T20:31:10.558+0000 7f3a1949ba40 -1 Falling back to public interface 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: 2026-02-22T20:31:11.210+0000 7f3a1949ba40 -1 osd.1 56 log_to_monitors true 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: 1 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: osd.1 up in weight 1 up_from 60 up_thru 60 down_at 57 last_clean_interval [49,56) [v2:127.0.0.1:6810/2200990452,v1:127.0.0.1:6812/2200990452] [v2:127.0.0.1:6814/2200990452,v1:127.0.0.1:6816/2200990452] exists,up 03c0b956-e207-4fd8-aad4-3d09bde31c58 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:31:11.550 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:31:11.551 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:31:11.551 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:31:11.551 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:31:11.551 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:31:11.551 INFO:tasks.workunit.client.0.trial150.stderr:155846: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:31:11.575 INFO:tasks.workunit.client.0.trial150.stderr:155846: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: g55848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-22T20:31:11.575 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:31:11.575 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:31:11.576 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:31:11.576 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:31:11.576 INFO:tasks.workunit.client.0.trial150.stderr:155848: 2026-02-22T20:31:10.098+0000 7f4a21717a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:11.576 INFO:tasks.workunit.client.0.trial150.stderr:155848: 2026-02-22T20:31:10.102+0000 7f4a21717a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:11.576 INFO:tasks.workunit.client.0.trial150.stderr:155848: 2026-02-22T20:31:10.102+0000 7f4a21717a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:11.576 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:31:11.576 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:11.576 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:31:11.576 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:11.576 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:11.576 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:11.577 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:11.577 INFO:tasks.workunit.client.0.trial150.stderr:155848: 0 2026-02-22T20:31:11.577 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:11.577 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:31:11.577 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:11.577 INFO:tasks.workunit.client.0.trial150.stderr:155848: 2026-02-22T20:31:10.558+0000 7f4a21717a40 -1 Falling back to public interface 2026-02-22T20:31:11.577 INFO:tasks.workunit.client.0.trial150.stderr:155848: 2026-02-22T20:31:11.210+0000 7f4a21717a40 -1 osd.2 56 log_to_monitors true 2026-02-22T20:31:11.577 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:11.577 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:11.577 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:11.577 INFO:tasks.workunit.client.0.trial150.stderr:155848: 1 2026-02-22T20:31:11.577 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:11.578 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:31:11.578 INFO:tasks.workunit.client.0.trial150.stderr:155848: osd.2 up in weight 1 up_from 60 up_thru 46 down_at 57 last_clean_interval [13,56) [v2:127.0.0.1:6811/3069436194,v1:127.0.0.1:6813/3069436194] [v2:127.0.0.1:6815/3069436194,v1:127.0.0.1:6817/3069436194] exists,up fc6e0473-9175-4301-9e11-fc7b02a1d330 2026-02-22T20:31:11.578 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:31:11.578 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:31:11.578 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:31:11.578 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:31:11.578 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:31:11.578 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:31:11.578 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:31:11.578 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:31:11.578 INFO:tasks.workunit.client.0.trial150.stderr:155848: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:31:12.090 INFO:tasks.workunit.client.0.trial150.stderr:155848: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:31:12.090 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:31:12.090 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:31:12.091 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:31:12.091 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:31:12.091 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:31:12.091 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:31:12.091 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:31:12.091 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:31:12.091 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:31:12.091 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:31:12.092 INFO:tasks.workunit.client.0.trial150.stderr:155846: 1 2026-02-22T20:31:12.092 INFO:tasks.workunit.client.0.trial150.stderr:155846: 2 2026-02-22T20:31:12.092 INFO:tasks.workunit.client.0.trial150.stderr:155846: 3' 2026-02-22T20:31:12.092 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:31:12.092 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:12.092 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:31:12.092 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-22T20:31:12.093 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-22T20:31:12.093 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-22T20:31:12.093 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:12.093 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:31:12.093 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037762 2026-02-22T20:31:12.093 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037762 2026-02-22T20:31:12.093 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-257698037762' 2026-02-22T20:31:12.093 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:12.093 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:31:12.094 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037762 2026-02-22T20:31:12.094 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037762 2026-02-22T20:31:12.094 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-257698037762 2-257698037762' 2026-02-22T20:31:12.094 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:12.094 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:31:12.094 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=236223201284 2026-02-22T20:31:12.094 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 236223201284 2026-02-22T20:31:12.094 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-257698037762 2-257698037762 3-236223201284' 2026-02-22T20:31:12.094 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:12.095 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/et_timeout_delays: echo true 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: 1 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: 2 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: 3' 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836500 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836500 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500' 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:31:12.127 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037763 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037763 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-257698037763' 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=257698037763 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 257698037763 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-257698037763 2-257698037763' 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=236223201285 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 236223201285 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500 1-257698037763 2-257698037763 3-236223201285' 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:12.128 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836500 2026-02-22T20:31:13.742 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:13.743 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:31:13.743 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-22T20:31:13.743 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:13.743 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-22T20:31:13.743 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-22T20:31:13.743 INFO:tasks.workunit.client.0.trial150.stderr:155846: waiting osd.0 seq 21474836499 2026-02-22T20:31:13.743 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:13.743 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836499 2026-02-22T20:31:13.743 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:31:13.743 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:31:13.743 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:13.744 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836499 2026-02-22T20:31:13.744 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:13.744 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-257698037762 2026-02-22T20:31:13.744 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:13.744 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:31:13.744 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-257698037762 2026-02-22T20:31:13.744 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:13.744 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037762 2026-02-22T20:31:13.744 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 257698037762' 2026-02-22T20:31:13.744 INFO:tasks.workunit.client.0.trial150.stderr:155846: waiting osd.1 seq 257698037762 2026-02-22T20:31:13.744 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:31:13.744 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037763 -lt 257698037762 2026-02-22T20:31:13.745 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:13.745 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-257698037762 2026-02-22T20:31:13.745 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:13.745 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:31:13.745 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-257698037762 2026-02-22T20:31:13.745 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:13.745 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037762 2026-02-22T20:31:13.745 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 257698037762' 2026-02-22T20:31:13.745 INFO:tasks.workunit.client.0.trial150.stderr:155846: waiting osd.2 seq 257698037762 2026-02-22T20:31:13.745 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:13.745 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037763 -lt 257698037762 2026-02-22T20:31:13.745 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836500 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836500 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: waiting osd.0 seq 21474836500 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836500 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836500 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-257698037763 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-257698037763 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:13.773 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037763 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 257698037763' 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: waiting osd.1 seq 257698037763 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037763 -lt 257698037763 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-257698037763 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-257698037763 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=257698037763 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 257698037763' 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: waiting osd.2 seq 257698037763 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 257698037763 -lt 257698037763 2026-02-22T20:31:13.774 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:14.460 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-236223201284 2026-02-22T20:31:14.460 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:14.460 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:31:14.461 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-236223201284 2026-02-22T20:31:14.461 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:14.461 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=236223201284 2026-02-22T20:31:14.461 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 236223201284' 2026-02-22T20:31:14.461 INFO:tasks.workunit.client.0.trial150.stderr:155846: waiting osd.3 seq 236223201284 2026-02-22T20:31:14.461 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:31:14.461 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 236223201285 -lt 236223201284 2026-02-22T20:31:14.461 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:31:14.461 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:14.461 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:14.461 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:31:14.462 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:31:14.463 INFO:tasks.workunit.client.0.trial150.stderr:155846: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:31:14.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:31:14.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:31:14.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 155845 2026-02-22T20:31:14.492 INFO:tasks.workunit.client.0.trial150.stderr::2267: flush_pg_stats: echo 3-236223201285 2026-02-22T20:31:14.493 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:14.493 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:31:14.493 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-236223201285 2026-02-22T20:31:14.493 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:14.493 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=236223201285 2026-02-22T20:31:14.493 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 236223201285' 2026-02-22T20:31:14.493 INFO:tasks.workunit.client.0.trial150.stderr:155848: waiting osd.3 seq 236223201285 2026-02-22T20:31:14.493 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:31:14.493 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 236223201285 -lt 236223201285 2026-02-22T20:31:14.494 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:31:14.494 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:14.494 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:14.494 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:31:14.494 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:14.494 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:14.494 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:14.494 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:14.494 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:14.494 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:14.494 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:14.494 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:31:14.495 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:14.495 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:14.495 INFO:tasks.workunit.client.0.trial150.stderr:155848: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:14.495 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:31:14.495 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:31:14.495 INFO:tasks.workunit.client.0.trial150.stderr:155848: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:31:14.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:31:14.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:31:14.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:31:14.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:31:14.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-22T20:31:14.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:31:14.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-22T20:31:14.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:31:14.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:31:14.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:31:14.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:31:14.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-22T20:31:14.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-22T20:31:14.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:31:14.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:31:14.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:14.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:14.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:14.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:14.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:30:59.399491+0000 2026-02-22T20:31:14.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:31:14.961 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:31:14.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:30:59.399491+0000 2026-02-22T20:31:14.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:31:14.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:30:59.399491+0000 2026-02-22T20:31:14.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:31:14.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:31:14.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:14.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:14.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:14.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:14.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:14.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:15.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:59.399491+0000 '>' 2026-02-22T20:30:59.399491+0000 2026-02-22T20:31:15.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:16.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:16.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:16.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:16.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:16.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:16.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:16.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:16.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:59.399491+0000 '>' 2026-02-22T20:30:59.399491+0000 2026-02-22T20:31:16.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:17.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:17.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:17.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:17.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:17.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:17.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:17.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:17.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:59.399491+0000 '>' 2026-02-22T20:30:59.399491+0000 2026-02-22T20:31:17.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:18.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:18.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:18.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:18.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:18.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:18.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:18.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:18.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:59.399491+0000 '>' 2026-02-22T20:30:59.399491+0000 2026-02-22T20:31:18.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:19.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:19.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:19.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:19.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:19.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:19.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:19.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:19.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:59.399491+0000 '>' 2026-02-22T20:30:59.399491+0000 2026-02-22T20:31:19.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:20.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:20.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:20.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:20.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:20.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:20.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:20.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:59.399491+0000 '>' 2026-02-22T20:30:59.399491+0000 2026-02-22T20:31:20.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:21.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:21.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:21.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:21.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:21.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:21.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:21.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:22.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:30:59.399491+0000 '>' 2026-02-22T20:30:59.399491+0000 2026-02-22T20:31:22.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:23.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:23.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:23.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:23.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:23.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:23.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:23.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:23.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:31:16.061653+0000 '>' 2026-02-22T20:30:59.399491+0000 2026-02-22T20:31:23.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:31:23.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-22T20:31:23.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:31:23.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:31:23.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:31:23.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 158730"' 2026-02-22T20:31:23.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 158730' 2026-02-22T20:31:23.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-22T20:31:23.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:31:23.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:31:23.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 158732"' 2026-02-22T20:31:23.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:31:23.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 158732' 2026-02-22T20:31:23.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-22T20:31:23.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/158733: /' 2026-02-22T20:31:23.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 158730 158732' 2026-02-22T20:31:23.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:31:23.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:31:23.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 158730 2026-02-22T20:31:23.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-22T20:31:23.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/158736: /' 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING list-attrs 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING list-attrs 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: _ 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: hinfo_key 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: snapset 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-22T20:31:23.753 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:23.754 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:31:23.754 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:31:23.754 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:31:23.754 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:31:23.754 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:23.754 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:31:23.756 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:31:23.757 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:31:23.757 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:31:23.757 INFO:tasks.workunit.client.0.trial150.stderr:158736: start osd.2 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:31:23.769 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: _ 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: hinfo_key 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: snapset 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:23.770 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:23.773 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:23.774 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:23.774 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:23.774 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:23.774 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:23.774 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:23.774 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:31:23.774 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:31:23.774 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:31:23.774 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:31:23.774 INFO:tasks.workunit.client.0.trial150.stderr:158733: start osd.1 2026-02-22T20:31:25.236 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: 2026-02-22T20:31:23.766+0000 7f95066d1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: 2026-02-22T20:31:23.766+0000 7f95066d1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: 2026-02-22T20:31:23.770+0000 7f95066d1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:25.237 INFO:tasks.workunit.client.0.trial150.stderr:158736: 0 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: 2026-02-22T20:31:24.194+0000 7f95066d1a40 -1 Falling back to public interface 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: 2026-02-22T20:31:24.822+0000 7f95066d1a40 -1 osd.2 63 log_to_monitors true 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: 1 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: osd.2 up in weight 1 up_from 68 up_thru 46 down_at 64 last_clean_interval [60,63) [v2:127.0.0.1:6818/939268472,v1:127.0.0.1:6819/939268472] [v2:127.0.0.1:6820/939268472,v1:127.0.0.1:6821/939268472] exists,up fc6e0473-9175-4301-9e11-fc7b02a1d330 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:31:25.238 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:31:25.239 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:31:25.239 INFO:tasks.workunit.client.0.trial150.stderr:158736: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:31:25.252 INFO:tasks.workunit.client.0.trial150.stderr:158736: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1very_ops 2026-02-22T20:31:25.252 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:31:25.252 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:31:25.252 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:31:25.252 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:31:25.252 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:31:25.252 INFO:tasks.workunit.client.0.trial150.stderr:158733: 2026-02-22T20:31:23.782+0000 7f30bf92ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: 2026-02-22T20:31:23.786+0000 7f30bf92ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: 2026-02-22T20:31:23.786+0000 7f30bf92ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: 0 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: 2026-02-22T20:31:24.130+0000 7f30bf92ea40 -1 Falling back to public interface 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: 2026-02-22T20:31:24.718+0000 7f30bf92ea40 -1 osd.1 63 log_to_monitors true 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: 1 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: osd.1 up in weight 1 up_from 67 up_thru 68 down_at 64 last_clean_interval [60,63) [v2:127.0.0.1:6810/3115350752,v1:127.0.0.1:6811/3115350752] [v2:127.0.0.1:6812/3115350752,v1:127.0.0.1:6813/3115350752] exists,up 03c0b956-e207-4fd8-aad4-3d09bde31c58 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:31:25.253 INFO:tasks.workunit.client.0.trial150.stderr:158733: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:31:25.785 INFO:tasks.workunit.client.0.trial150.stderr:158733: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.593: get_timeout_delays: echo true 2026-02-22T20:31:25.786 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:31:25.786 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:31:25.786 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:31:25.786 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:31:25.786 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:31:25.786 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:31:25.786 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:31:25.786 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:31:25.787 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:31:25.787 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:31:25.787 INFO:tasks.workunit.client.0.trial150.stderr:158736: 1 2026-02-22T20:31:25.787 INFO:tasks.workunit.client.0.trial150.stderr:158736: 2 2026-02-22T20:31:25.787 INFO:tasks.workunit.client.0.trial150.stderr:158736: 3' 2026-02-22T20:31:25.787 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:31:25.787 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:25.787 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:31:25.787 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836504 2026-02-22T20:31:25.787 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836504 2026-02-22T20:31:25.787 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504' 2026-02-22T20:31:25.787 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:25.788 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:31:25.788 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808834 2026-02-22T20:31:25.788 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808834 2026-02-22T20:31:25.788 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-287762808834' 2026-02-22T20:31:25.788 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:25.788 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:31:25.788 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776130 2026-02-22T20:31:25.788 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776130 2026-02-22T20:31:25.788 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-287762808834 2-292057776130' 2026-02-22T20:31:25.788 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:25.788 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:31:25.788 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=236223201288 2026-02-22T20:31:25.789 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 236223201288 2026-02-22T20:31:25.789 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-287762808834 2-292057776130 3-236223201288' 2026-02-22T20:31:25.789 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:25.789 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836504 2026-02-22T20:31:25.807 INFO:tasks.workunit.client.0.trial150.stderr:158736: /sh:1593: get_timeout_delays: echo true 2026-02-22T20:31:25.807 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:31:25.807 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:31:25.807 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:31:25.807 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:31:25.807 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:31:25.807 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:31:25.807 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:31:25.808 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:31:25.808 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:31:25.808 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:31:25.808 INFO:tasks.workunit.client.0.trial150.stderr:158733: 1 2026-02-22T20:31:25.808 INFO:tasks.workunit.client.0.trial150.stderr:158733: 2 2026-02-22T20:31:25.808 INFO:tasks.workunit.client.0.trial150.stderr:158733: 3' 2026-02-22T20:31:25.808 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:31:25.808 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:25.808 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:31:25.808 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836505 2026-02-22T20:31:25.808 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836505 2026-02-22T20:31:25.808 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505' 2026-02-22T20:31:25.809 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:25.809 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:31:25.809 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808835 2026-02-22T20:31:25.809 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808835 2026-02-22T20:31:25.809 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-287762808835' 2026-02-22T20:31:25.809 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:25.809 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:31:25.809 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776131 2026-02-22T20:31:25.809 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776131 2026-02-22T20:31:25.810 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-287762808835 2-292057776131' 2026-02-22T20:31:25.810 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:25.810 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:31:25.810 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=236223201289 2026-02-22T20:31:25.810 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 236223201289 2026-02-22T20:31:25.810 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-287762808835 2-292057776131 3-236223201289' 2026-02-22T20:31:25.810 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:25.810 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836505 2026-02-22T20:31:27.451 INFO:tasks.workunit.client.0.trial150.stderr:15873/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:27.451 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:31:27.451 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836504 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836504 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: waiting osd.0 seq 21474836504 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836504 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836504 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-287762808834 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-287762808834 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808834 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: waiting osd.1 seq 287762808834 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808835 -lt 287762808834 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776130 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776130 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776130 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776130' 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: waiting osd.2 seq 292057776130 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776131 -lt 292057776130 2026-02-22T20:31:27.452 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help3: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836505 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836505 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: waiting osd.0 seq 21474836505 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836505 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836505 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-287762808835 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-287762808835 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808835 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 287762808835' 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: waiting osd.1 seq 287762808835 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808835 -lt 287762808835 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776131 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776131 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776131 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776131' 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: waiting osd.2 seq 292057776131 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776131 -lt 292057776131 2026-02-22T20:31:27.455 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:28.269 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-ers.sh:2267: flush_pg_stats: echo 3-236223201288 2026-02-22T20:31:28.269 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:28.269 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:31:28.269 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-236223201288 2026-02-22T20:31:28.269 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:28.269 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=236223201288 2026-02-22T20:31:28.269 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 236223201288' 2026-02-22T20:31:28.269 INFO:tasks.workunit.client.0.trial150.stderr:158736: waiting osd.3 seq 236223201288 2026-02-22T20:31:28.269 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 236223201290 -lt 236223201288 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' -1 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=3 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:28.270 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local helpers.sh:2267: flush_pg_stats: echo 3-236223201289 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-236223201289 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=236223201289 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 236223201289' 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158733: waiting osd.3 seq 236223201289 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 236223201290 -lt 236223201289 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:28.271 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' -1 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=3 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:28.272 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:31:29.036 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:29.037 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: getexpression 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:31:29.040 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:31:29.041 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:31:29.041 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:31:29.041 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:31:29.041 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:29.041 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:29.041 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:29.041 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:29.041 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:29.041 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:30.148 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:30.148 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-22T20:31:30.149 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:30.149 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:30.149 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:30.149 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-22T20:31:30.149 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-22T20:31:30.149 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:31:30.149 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:31:30.149 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:31:30.149 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:31:30.149 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:31:30.150 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:31:30.150 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:31:30.150 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:31:30.150 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:31:30.150 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-22T20:31:30.150 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:31:30.150 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-22T20:31:30.150 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:31:30.150 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:30.150 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:30.150 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:30.150 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:30.151 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:30.151 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:30.151 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:30.151 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-22T20:31:30.151 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:30.151 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:30.158 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-h_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:30.158 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-22T20:31:30.158 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:30.158 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:30.158 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:30.159 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-22T20:31:30.159 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-22T20:31:30.159 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:31:30.159 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:31:30.159 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:31:30.159 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:31:30.159 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:31:30.159 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:31:30.159 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:31:30.159 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:31:30.159 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:31:30.159 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-22T20:31:30.160 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:31:30.160 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-22T20:31:30.160 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:31:30.160 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:30.160 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:30.160 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:30.160 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:30.160 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:30.160 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:30.160 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:30.160 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-22T20:31:30.160 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:30.161 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:31.700 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceelpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:31.700 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-22T20:31:31.700 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-22T20:31:31.701 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:31:31.701 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:31:31.701 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:31:31.701 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:31:31.701 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:31:31.701 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:31:31.701 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:31:31.701 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:31:31.701 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:31:31.701 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-22T20:31:31.702 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:31:31.702 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-22T20:31:31.702 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:31:31.702 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:31.702 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:31.702 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:31.702 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:31.702 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:31.702 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:31.702 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:31.703 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:31:31.703 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:31.703 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:31.703 INFO:tasks.workunit.client.0.trial150.stderr:158736: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:31.703 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:31:31.703 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:31:31.703 INFO:tasks.workunit.client.0.trial150.stderr:158736: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:31:31.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:31:31.704 INFO:tasks.workunit.client.0.trial150.stderr:ph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:31.705 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 5 2026-02-22T20:31:31.705 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 3 '!=' 3 2026-02-22T20:31:31.705 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:31:31.705 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:31:31.705 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:31:31.705 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:31:31.705 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:31:31.705 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:31:31.706 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:31:31.706 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:31:31.706 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:31:31.706 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-22T20:31:31.706 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:31:31.706 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-22T20:31:31.706 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:31:31.706 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:31.706 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:31.706 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:31.706 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:31.706 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:158733: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:158733: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 158732 2026-02-22T20:31:31.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:31:31.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:31:31.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:31:31.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-22T20:31:31.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:31:31.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:31:31.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:31:31.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 1 2026-02-22T20:31:31.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-22T20:31:31.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-22T20:31:31.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=3 2026-02-22T20:31:31.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=1 2026-02-22T20:31:31.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-22T20:31:31.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:31:31.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:31:31.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:31:31.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 161539"' 2026-02-22T20:31:31.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 161539' 2026-02-22T20:31:31.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:31:31.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:31:31.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:31:31.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 161540"' 2026-02-22T20:31:31.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 161540' 2026-02-22T20:31:31.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-22T20:31:31.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 161539 161540' 2026-02-22T20:31:31.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:31:31.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:31:31.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 161539 2026-02-22T20:31:31.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:31:31.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/161542: /' 2026-02-22T20:31:31.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:31:31.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/161544: /' 2026-02-22T20:31:32.334 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:31:32.334 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:31:32.334 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:31:32.334 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:31:32.334 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:31:32.335 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:31:32.335 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:31:32.335 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:31:32.335 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:31:32.335 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:31:32.335 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:31:32.335 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:31:32.335 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:31:32.335 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:31:32.335 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:31:32.335 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:31:32.335 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: remove 0#2:eb822e21:::SOMETHING:head# 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:32.336 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:32.337 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:32.337 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:32.337 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:32.338 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:32.339 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:32.339 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:32.339 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:32.339 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:32.339 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:31:32.339 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:31:32.339 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:31:32.339 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:31:32.339 INFO:tasks.workunit.client.0.trial150.stderr:161542: start osd.3 2026-02-22T20:31:32.339 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:31:32.354 INFO:tasks.workunit.client.0.trial150.stderr:1161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:31:32.354 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:31:32.354 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:31:32.354 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:31:32.354 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:31:32.354 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:31:32.354 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:31:32.354 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:31:32.354 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:31:32.355 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-22T20:31:32.356 INFO:tasks.workunit.client.0.trial150.stderr:161544: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:31:32.356 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:31:32.356 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:32.356 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:31:32.356 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:31:32.356 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:31:32.356 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:31:32.356 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:32.356 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:32.357 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:32.357 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:32.357 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:31:32.357 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:31:32.357 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:32.357 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:31:32.357 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:32.357 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:31:32.357 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:32.357 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:31:32.358 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:31:32.359 INFO:tasks.workunit.client.0.trial150.stderr:161544: start osd.1 2026-02-22T20:31:32.359 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: 2026-02-22T20:31:32.346+0000 7fb8464ffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: 2026-02-22T20:31:32.350+0000 7fb8464ffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: 2026-02-22T20:31:32.350+0000 7fb8464ffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: 0 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: 2026-02-22T20:31:32.822+0000 7fb8464ffa40 -1 Falling back to public interface 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: 2026-02-22T20:31:33.550+0000 7fb8464ffa40 -1 osd.3 69 log_to_monitors true 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:33.826 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: 1 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: osd.3 up in weight 1 up_from 73 up_thru 68 down_at 70 last_clean_interval [55,69) [v2:127.0.0.1:6810/4199436001,v1:127.0.0.1:6811/4199436001] [v2:127.0.0.1:6812/4199436001,v1:127.0.0.1:6813/4199436001] exists,up 68dadeef-d802-40cc-aa88-89d20bdaf7d9 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:31:33.827 INFO:tasks.workunit.client.0.trial150.stderr:161542: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161542: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: g61544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: 2026-02-22T20:31:32.362+0000 7f85da595a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: 2026-02-22T20:31:32.362+0000 7f85da595a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: 2026-02-22T20:31:32.366+0000 7f85da595a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: 0 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: 2026-02-22T20:31:32.838+0000 7f85da595a40 -1 Falling back to public interface 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: 2026-02-22T20:31:33.550+0000 7f85da595a40 -1 osd.1 69 log_to_monitors true 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: 1 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: osd.1 up in weight 1 up_from 73 up_thru 68 down_at 70 last_clean_interval [67,69) [v2:127.0.0.1:6826/1579379636,v1:127.0.0.1:6827/1579379636] [v2:127.0.0.1:6828/1579379636,v1:127.0.0.1:6829/1579379636] exists,up 03c0b956-e207-4fd8-aad4-3d09bde31c58 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:31:33.828 INFO:tasks.workunit.client.0.trial150.stderr:161544: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:31:34.382 INFO:tasks.workunit.client.0.trial150.stderr:161544: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:31:34.382 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:31:34.382 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:31:34.382 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:31:34.382 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:31:34.382 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:31:34.382 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:31:34.382 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:31:34.382 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: 1 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: 2 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: 3' 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836507 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836507 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507' 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612610 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612610 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-313532612610' 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776134 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776134 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-313532612610 2-292057776134' 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612611 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612611 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836507 1-313532612610 2-292057776134 3-313532612611' 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:34.383 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836507 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/et_timeout_delays: echo true 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: 1 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: 2 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: 3' 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836508 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836508 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508' 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612611 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612611 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-313532612611' 2026-02-22T20:31:34.384 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:34.385 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:31:34.385 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776133 2026-02-22T20:31:34.385 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776133 2026-02-22T20:31:34.385 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-313532612611 2-292057776133' 2026-02-22T20:31:34.385 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:34.385 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:31:34.385 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612610 2026-02-22T20:31:34.385 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612610 2026-02-22T20:31:34.385 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836508 1-313532612611 2-292057776133 3-313532612610' 2026-02-22T20:31:34.385 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:34.385 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836508 2026-02-22T20:31:37.042 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:37.042 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:31:37.042 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836507 2026-02-22T20:31:37.042 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:37.042 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836507 2026-02-22T20:31:37.042 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: waiting osd.0 seq 21474836507 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836507 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-313532612610 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-313532612610 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612610 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 313532612610' 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: waiting osd.1 seq 313532612610 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612611 -lt 313532612610 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776134 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776134 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776134 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776134' 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: waiting osd.2 seq 292057776134 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776132 -lt 292057776134 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776132 -lt 292057776134 2026-02-22T20:31:37.043 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272:ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836508 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836508 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: waiting osd.0 seq 21474836508 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836508 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-313532612611 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-313532612611 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612611 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 313532612611' 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: waiting osd.1 seq 313532612611 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612611 -lt 313532612611 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776133 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776133 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776133 2026-02-22T20:31:37.044 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776133' 2026-02-22T20:31:37.045 INFO:tasks.workunit.client.0.trial150.stderr:161544: waiting osd.2 seq 292057776133 2026-02-22T20:31:37.045 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:37.045 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776132 -lt 292057776133 2026-02-22T20:31:37.045 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:31:37.045 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:31:37.045 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:37.045 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776132 -lt 292057776133 2026-02-22T20:31:37.045 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:31:37.905 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:31:37.905 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:37.905 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776135 -lt 292057776133 2026-02-22T20:31:37.905 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:37.905 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-313532612610 2026-02-22T20:31:37.905 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:37.905 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:31:37.905 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-313532612610 2026-02-22T20:31:37.905 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:37.906 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612610 2026-02-22T20:31:37.906 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 313532612610' 2026-02-22T20:31:37.906 INFO:tasks.workunit.client.0.trial150.stderr:161544: waiting osd.3 seq 313532612610 2026-02-22T20:31:37.906 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:31:37.906 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612611 -lt 313532612610 2026-02-22T20:31:37.906 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:31:37.906 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:37.906 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:37.906 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:31:37.906 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:37.906 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:37.906 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:37.907 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:37.907 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:37.907 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:37.907 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:37.907 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:31:37.907 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:37.907 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:37.907 INFO:tasks.workunit.client.0.trial150.stderr:161544: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:37.907 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:31:37.907 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:31:37.907 INFO:tasks.workunit.client.0.trial150.stderr:161544: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:31:37.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:31:37.920 INFO:tasks.workunit.client.0.trial150.stderr: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:31:37.920 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:37.920 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776135 -lt 292057776134 2026-02-22T20:31:37.920 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:37.920 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-313532612611 2026-02-22T20:31:37.921 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:37.921 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:31:37.921 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-313532612611 2026-02-22T20:31:37.921 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:37.921 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612611 2026-02-22T20:31:37.921 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 313532612611' 2026-02-22T20:31:37.921 INFO:tasks.workunit.client.0.trial150.stderr:161542: waiting osd.3 seq 313532612611 2026-02-22T20:31:37.921 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:31:37.921 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612611 -lt 313532612611 2026-02-22T20:31:37.921 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:31:37.921 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:37.922 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:37.922 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:31:37.922 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:37.922 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:37.922 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:37.922 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:37.922 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:37.922 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:37.922 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:37.922 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:31:37.922 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:37.922 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:37.923 INFO:tasks.workunit.client.0.trial150.stderr:161542: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:37.923 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:31:37.923 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:31:37.923 INFO:tasks.workunit.client.0.trial150.stderr:161542: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:31:37.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:31:37.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:31:37.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 161540 2026-02-22T20:31:37.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:31:37.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:31:37.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:31:37.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-22T20:31:37.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:31:37.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-22T20:31:37.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:31:37.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:31:37.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:31:37.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:31:38.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-22T20:31:38.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-22T20:31:38.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:31:38.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:31:38.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:38.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:38.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:38.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:38.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:31:16.061653+0000 2026-02-22T20:31:38.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:31:38.393 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:31:38.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:31:16.061653+0000 2026-02-22T20:31:38.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:31:38.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:31:16.061653+0000 2026-02-22T20:31:38.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:31:38.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:31:38.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:38.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:38.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:38.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:38.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:38.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:38.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:31:16.061653+0000 '>' 2026-02-22T20:31:16.061653+0000 2026-02-22T20:31:38.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:39.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:39.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:39.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:39.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:39.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:39.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:39.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:39.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:31:16.061653+0000 '>' 2026-02-22T20:31:16.061653+0000 2026-02-22T20:31:39.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:31:40.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:31:40.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:31:40.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:31:40.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:31:40.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:31:40.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:31:40.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:31:40.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:31:38.446450+0000 '>' 2026-02-22T20:31:16.061653+0000 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 164347"' 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 164347' 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 164349"' 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 164349' 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-22T20:31:40.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 164347 164349' 2026-02-22T20:31:40.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:31:40.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:31:40.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 164347 2026-02-22T20:31:40.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/164350: /' 2026-02-22T20:31:40.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:31:40.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/164352: /' 2026-02-22T20:31:41.397 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:31:41.397 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:31:41.397 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:31:41.397 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:31:41.397 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:31:41.397 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:31:41.397 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:31:41.397 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:31:41.397 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:31:41.397 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: _ 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: hinfo_key 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: snapset 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:41.398 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:31:41.399 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:31:41.399 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:31:41.399 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:31:41.399 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:41.399 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:41.400 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:41.400 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:41.400 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:31:41.400 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:31:41.400 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:41.400 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:31:41.401 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:41.401 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:31:41.401 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:41.401 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:41.401 INFO:tasks.workunit.client.0.trial150.stderr:164352: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:41.401 INFO:tasks.workunit.client.0.trial150.stderr:164352: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:41.401 INFO:tasks.workunit.client.0.trial150.stderr:164352: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:41.401 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:41.401 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:41.401 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:41.401 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:41.401 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:41.402 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:41.402 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:41.402 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:41.402 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:41.402 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:31:41.402 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:31:41.402 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:31:41.402 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:31:41.402 INFO:tasks.workunit.client.0.trial150.stderr:164352: start osd.1 2026-02-22T20:31:41.458 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:31:41.458 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:31:41.458 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:31:41.458 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:31:41.458 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:31:41.458 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:31:41.458 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:31:41.458 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:31:41.459 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:31:41.459 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:31:41.459 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:31:41.459 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:31:41.459 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:31:41.459 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:31:41.459 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:31:41.459 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:31:41.459 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:31:41.459 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:31:41.459 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:31:41.459 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:31:41.460 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:31:41.460 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:31:41.460 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-22T20:31:41.460 INFO:tasks.workunit.client.0.trial150.stderr:164350: Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-22T20:31:41.460 INFO:tasks.workunit.client.0.trial150.stderr:164350: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-22T20:31:41.460 INFO:tasks.workunit.client.0.trial150.stderr:164350: _ 2026-02-22T20:31:41.460 INFO:tasks.workunit.client.0.trial150.stderr:164350: hinfo_key 2026-02-22T20:31:41.460 INFO:tasks.workunit.client.0.trial150.stderr:164350: snapset 2026-02-22T20:31:41.460 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:31:41.460 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:41.460 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:31:41.460 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:31:41.461 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:31:41.461 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:31:41.461 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:41.461 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:41.461 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:41.461 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:41.462 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:41.463 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:41.463 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:41.463 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:41.463 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:41.463 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:41.463 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:41.463 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:41.463 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:31:41.463 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:31:41.463 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:31:41.463 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:31:41.463 INFO:tasks.workunit.client.0.trial150.stderr:164350: start osd.3 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.very_ops 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164352: 2026-02-22T20:31:41.410+0000 7ffb5ec4fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164352: 2026-02-22T20:31:41.410+0000 7ffb5ec4fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164352: 2026-02-22T20:31:41.410+0000 7ffb5ec4fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:31:42.893 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: 0 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: 2026-02-22T20:31:41.878+0000 7ffb5ec4fa40 -1 Falling back to public interface 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: 2026-02-22T20:31:42.506+0000 7ffb5ec4fa40 -1 osd.1 75 log_to_monitors true 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:42.894 INFO:tasks.workunit.client.0.trial150.stderr:164352: 1 2026-02-22T20:31:42.895 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:42.895 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:31:42.895 INFO:tasks.workunit.client.0.trial150.stderr:164352: osd.1 up in weight 1 up_from 79 up_thru 79 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6810/184537080,v1:127.0.0.1:6811/184537080] [v2:127.0.0.1:6812/184537080,v1:127.0.0.1:6813/184537080] exists,up 03c0b956-e207-4fd8-aad4-3d09bde31c58 2026-02-22T20:31:42.895 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:31:42.895 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:31:42.895 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:31:42.895 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:31:42.895 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:31:42.895 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:31:42.895 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:31:42.895 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:31:42.895 INFO:tasks.workunit.client.0.trial150.stderr:164352: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:31:42.953 INFO:tasks.workunit.client.0.trial150.stderr:164352: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:31:42.953 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:31:42.953 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:31:42.953 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:31:42.953 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:31:42.953 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: 2026-02-22T20:31:41.470+0000 7f11849d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: 2026-02-22T20:31:41.470+0000 7f11849d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: 2026-02-22T20:31:41.474+0000 7f11849d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: 0 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: 2026-02-22T20:31:41.934+0000 7f11849d2a40 -1 Falling back to public interface 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: 2026-02-22T20:31:42.594+0000 7f11849d2a40 -1 osd.3 75 log_to_monitors true 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: 1 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: osd.3 up in weight 1 up_from 80 up_thru 80 down_at 76 last_clean_interval [73,75) [v2:127.0.0.1:6826/3414163237,v1:127.0.0.1:6827/3414163237] [v2:127.0.0.1:6828/3414163237,v1:127.0.0.1:6829/3414163237] exists,up 68dadeef-d802-40cc-aa88-89d20bdaf7d9 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:31:42.954 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:31:42.955 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:31:43.439 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_del593: get_timeout_delays: echo true 2026-02-22T20:31:43.439 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:31:43.439 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: 1 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: 2 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: 3' 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836511 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836511 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511' 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=339302416386 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 339302416386 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-339302416386' 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776137 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776137 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-339302416386 2-292057776137' 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383682 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383682 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-339302416386 2-292057776137 3-343597383682' 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:43.440 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836511 2026-02-22T20:31:43.511 INFO:tasks.workunit.client.0.trial150.stderr:164352: /ays 90 .1 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: 1 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: 2 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: 3' 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836512 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836512 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512' 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=339302416387 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 339302416387 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-339302416387' 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=292057776138 2026-02-22T20:31:43.512 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 292057776138 2026-02-22T20:31:43.513 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-339302416387 2-292057776138' 2026-02-22T20:31:43.513 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:31:43.513 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:31:43.513 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383683 2026-02-22T20:31:43.513 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383683 2026-02-22T20:31:43.513 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836512 1-339302416387 2-292057776138 3-343597383683' 2026-02-22T20:31:45.093 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:45.094 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:31:45.094 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836511 2026-02-22T20:31:45.094 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:45.094 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836511 2026-02-22T20:31:45.094 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2026-02-22T20:31:45.094 INFO:tasks.workunit.client.0.trial150.stderr:164352: waiting osd.0 seq 21474836511 2026-02-22T20:31:45.094 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:45.094 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836511 2026-02-22T20:31:45.094 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:31:45.094 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:31:45.094 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:45.095 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836511 2026-02-22T20:31:45.095 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:45.095 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-339302416386 2026-02-22T20:31:45.095 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:45.095 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:31:45.095 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-339302416386 2026-02-22T20:31:45.095 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:45.095 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=339302416386 2026-02-22T20:31:45.095 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 339302416386' 2026-02-22T20:31:45.095 INFO:tasks.workunit.client.0.trial150.stderr:164352: waiting osd.1 seq 339302416386 2026-02-22T20:31:45.095 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:31:45.095 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 339302416387 -lt 339302416386 2026-02-22T20:31:45.096 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:45.096 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776137 2026-02-22T20:31:45.096 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:45.096 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:31:45.096 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776137 2026-02-22T20:31:45.096 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:45.096 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776137 2026-02-22T20:31:45.096 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776137' 2026-02-22T20:31:45.096 INFO:tasks.workunit.client.0.trial150.stderr:164352: waiting osd.2 seq 292057776137 2026-02-22T20:31:45.096 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:45.096 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 292057776138 -lt 292057776137 2026-02-22T20:31:45.096 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:45.153 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-help/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:45.153 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836512 2026-02-22T20:31:45.153 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:45.153 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:31:45.153 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836512 2026-02-22T20:31:45.153 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:45.153 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836512 2026-02-22T20:31:45.153 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2026-02-22T20:31:45.153 INFO:tasks.workunit.client.0.trial150.stderr:164350: waiting osd.0 seq 21474836512 2026-02-22T20:31:45.154 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:45.154 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836512 2026-02-22T20:31:45.154 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:31:45.154 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:31:45.154 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:31:45.154 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836512 2026-02-22T20:31:45.154 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:45.154 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-339302416387 2026-02-22T20:31:45.154 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:45.154 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:31:45.154 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-339302416387 2026-02-22T20:31:45.154 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:45.155 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=339302416387 2026-02-22T20:31:45.155 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 339302416387' 2026-02-22T20:31:45.155 INFO:tasks.workunit.client.0.trial150.stderr:164350: waiting osd.1 seq 339302416387 2026-02-22T20:31:45.155 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:31:45.155 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 339302416387 -lt 339302416387 2026-02-22T20:31:45.155 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:45.155 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-292057776138 2026-02-22T20:31:45.155 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:45.155 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:31:45.155 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-292057776138 2026-02-22T20:31:45.155 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:45.155 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=292057776138 2026-02-22T20:31:45.156 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 292057776138' 2026-02-22T20:31:45.156 INFO:tasks.workunit.client.0.trial150.stderr:164350: waiting osd.2 seq 292057776138 2026-02-22T20:31:45.156 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:31:45.812 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 2ers.sh:2267: flush_pg_stats: echo 3-343597383682 2026-02-22T20:31:45.812 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:45.812 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:31:45.812 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-343597383682 2026-02-22T20:31:45.812 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:45.812 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383682 2026-02-22T20:31:45.813 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 343597383682' 2026-02-22T20:31:45.813 INFO:tasks.workunit.client.0.trial150.stderr:164352: waiting osd.3 seq 343597383682 2026-02-22T20:31:45.813 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:31:45.813 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383682 2026-02-22T20:31:45.813 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:31:45.813 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:45.813 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:45.813 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:31:45.813 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:45.813 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:45.813 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:45.813 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:45.814 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:45.814 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:45.814 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:45.814 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:31:45.814 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:45.814 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:45.814 INFO:tasks.workunit.client.0.trial150.stderr:164352: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:45.814 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:31:45.814 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:31:45.814 INFO:tasks.workunit.client.0.trial150.stderr:164352: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:31:45.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:31:45.857 INFO:tasks.workunit.client.0.trial150.stderr:92057776138 -lt 292057776138 2026-02-22T20:31:45.858 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:31:45.858 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-343597383683 2026-02-22T20:31:45.858 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:31:45.858 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:31:45.858 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-343597383683 2026-02-22T20:31:45.858 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:31:45.858 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383683 2026-02-22T20:31:45.858 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 343597383683' 2026-02-22T20:31:45.858 INFO:tasks.workunit.client.0.trial150.stderr:164350: waiting osd.3 seq 343597383683 2026-02-22T20:31:45.859 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:31:45.859 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383683 -lt 343597383683 2026-02-22T20:31:45.859 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:31:45.859 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:45.859 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:45.859 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:31:45.859 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:31:45.859 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:31:45.859 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:31:45.859 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:31:45.859 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:31:45.859 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:31:45.860 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:31:45.860 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:31:45.860 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:31:45.860 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:31:45.860 INFO:tasks.workunit.client.0.trial150.stderr:164350: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:31:45.860 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:31:45.860 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:31:45.860 INFO:tasks.workunit.client.0.trial150.stderr:164350: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:31:45.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:31:45.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:31:45.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 164349 2026-02-22T20:31:45.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:31:45.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:31:45.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:31:45.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-22T20:31:45.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:31:45.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:31:45.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:31:45.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:31:45.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:31:45.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:31:45.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:31:45.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:31:45.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:31:45.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:31:45.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:31:45.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:31:45.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:31:45.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:31:45.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:31:45.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:31:45.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:31:45.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:31:45.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:31:45.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:31:45.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:31:45.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:31:45.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:31:45.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:31:45.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:31:45.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:31:46.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:31:46.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:46.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:46.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:31:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:31:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:31:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:31:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:31:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:31:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:31:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:31:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:31:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:31:46.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:31:46.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:31:46.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:31:46.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:31:46.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:31:46.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:31:46.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:31:46.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:31:46.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:31:46.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:31:46.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:31:46.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:31:46.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:31:46.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:31:46.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:31:46.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:31:46.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:31:46.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:31:46.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:31:46.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:31:46.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:46.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:46.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:31:46.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:31:46.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:31:46.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:31:46.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:31:46.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:46.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:46.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:31:46.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:31:46.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:31:46.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:31:46.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:31:46.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_appends td/osd-scrub-repair 2026-02-22T20:31:46.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:901: TEST_corrupt_and_repair_lrc_appends: corrupt_and_repair_lrc td/osd-scrub-repair false 2026-02-22T20:31:46.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:884: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2026-02-22T20:31:46.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:885: corrupt_and_repair_lrc: local allow_overwrites=false 2026-02-22T20:31:46.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:886: corrupt_and_repair_lrc: local poolname=ecpool 2026-02-22T20:31:46.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2026-02-22T20:31:46.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:31:46.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:31:46.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:31:46.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:31:46.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:31:46.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:31:46.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:31:46.069 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:46.069 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:46.069 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:46.069 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:46.069 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:46.069 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:46.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:31:46.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:31:46.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:31:46.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:31:46.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:31:46.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:31:46.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:31:46.097 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:31:46.097 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:31:46.097 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:31:46.098 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:31:46.098 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:46.098 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:46.098 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:31:46.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:31:46.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:31:46.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:31:46.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:31:46.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:31:46.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:31:46.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:31:46.156 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:31:46.156 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:31:46.156 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:31:46.157 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:31:46.157 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:46.157 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:46.157 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:31:46.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:31:46.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:31:46.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:889: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2026-02-22T20:31:46.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:31:46.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:31:46.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:31:46.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:31:46.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:31:46.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:31:46.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:31:46.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:46.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:46.307 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:46.307 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:46.307 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:46.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:46.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:31:46.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:31:46.327 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: seq 0 9 2026-02-22T20:31:46.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:31:46.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2026-02-22T20:31:46.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:46.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:31:46.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:31:46.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:31:46.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:31:46.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:46.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:46.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:46.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:46.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:31:46.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:31:46.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:46.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:31:46.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:46.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:31:46.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:46.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:46.331 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:46.331 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:46.332 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:46.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:46.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:46.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:46.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:31:46.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:31:46.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:46.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:46.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:46.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:46.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:46.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:46.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:31:46.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:31:46.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:31:46.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:31:46.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=64534b75-b3dc-4fb0-b4ea-ae1f7e1a23e3 2026-02-22T20:31:46.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 64534b75-b3dc-4fb0-b4ea-ae1f7e1a23e3' 2026-02-22T20:31:46.335 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 64534b75-b3dc-4fb0-b4ea-ae1f7e1a23e3 2026-02-22T20:31:46.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:31:46.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCyZ5tp5dETFBAAPjPYNapg3KpII/364rtgAg== 2026-02-22T20:31:46.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCyZ5tp5dETFBAAPjPYNapg3KpII/364rtgAg=="}' 2026-02-22T20:31:46.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 64534b75-b3dc-4fb0-b4ea-ae1f7e1a23e3 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:31:46.432 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:31:46.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:31:46.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyZ5tp5dETFBAAPjPYNapg3KpII/364rtgAg== --osd-uuid 64534b75-b3dc-4fb0-b4ea-ae1f7e1a23e3 2026-02-22T20:31:46.457 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:46.445+0000 7f9de40b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:46.458 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:46.449+0000 7f9de40b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:46.459 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:46.449+0000 7f9de40b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:46.460 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:46.449+0000 7f9de40b7a40 -1 bdev(0x55d112c76c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:31:46.460 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:46.449+0000 7f9de40b7a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:31:47.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:31:47.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:31:47.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:31:47.243 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:31:47.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:31:47.450 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:31:47.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:31:47.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:31:47.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:31:47.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:31:47.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:31:47.474 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:47.465+0000 7fa34adeea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:47.476 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:47.465+0000 7fa34adeea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:47.477 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:47.465+0000 7fa34adeea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:47.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:31:47.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:47.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:31:47.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:47.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:47.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:47.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:47.609 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:31:47.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:47.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:31:47.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:47.871 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:47.861+0000 7fa34adeea40 -1 Falling back to public interface 2026-02-22T20:31:48.536 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:48.525+0000 7fa34adeea40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:31:48.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:48.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:48.770 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:31:48.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:48.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:48.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:31:48.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:49.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:49.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:49.937 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:31:49.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:31:49.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:49.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:31:50.096 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/121305966,v1:127.0.0.1:6803/121305966] [v2:127.0.0.1:6804/121305966,v1:127.0.0.1:6805/121305966] exists,up 64534b75-b3dc-4fb0-b4ea-ae1f7e1a23e3 2026-02-22T20:31:50.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:31:50.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:31:50.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:31:50.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:31:50.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2026-02-22T20:31:50.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:50.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:31:50.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:31:50.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:31:50.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:31:50.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:50.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:50.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:50.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:50.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:31:50.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:31:50.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:50.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:31:50.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:50.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:31:50.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:50.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:50.102 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:50.102 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:50.102 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:50.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:50.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:50.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:50.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:31:50.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:31:50.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:50.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:50.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:50.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:50.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:50.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:50.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:31:50.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:31:50.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:31:50.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:31:50.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c1dca8fe-ebc0-49e3-b0ec-1147c504e603 2026-02-22T20:31:50.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 c1dca8fe-ebc0-49e3-b0ec-1147c504e603' 2026-02-22T20:31:50.105 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 c1dca8fe-ebc0-49e3-b0ec-1147c504e603 2026-02-22T20:31:50.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:31:50.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC2Z5tpqOBRBhAAbqMc+opsvoC43itJ2AAEWQ== 2026-02-22T20:31:50.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC2Z5tpqOBRBhAAbqMc+opsvoC43itJ2AAEWQ=="}' 2026-02-22T20:31:50.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c1dca8fe-ebc0-49e3-b0ec-1147c504e603 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:31:50.266 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:31:50.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:31:50.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2Z5tpqOBRBhAAbqMc+opsvoC43itJ2AAEWQ== --osd-uuid c1dca8fe-ebc0-49e3-b0ec-1147c504e603 2026-02-22T20:31:50.292 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:50.281+0000 7fe4dd091a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:50.294 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:50.285+0000 7fe4dd091a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:50.294 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:50.285+0000 7fe4dd091a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:50.295 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:50.285+0000 7fe4dd091a40 -1 bdev(0x56183cbdec00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:31:50.295 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:50.285+0000 7fe4dd091a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:31:50.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:31:50.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:31:50.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:31:50.922 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:31:50.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:31:51.122 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:31:51.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:31:51.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:31:51.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:31:51.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:31:51.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:31:51.147 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:51.137+0000 7fcd9beb7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:51.149 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:51.137+0000 7fcd9beb7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:51.150 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:51.141+0000 7fcd9beb7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:51.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:31:51.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:51.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:31:51.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:51.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:51.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:51.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:51.286 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:31:51.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:51.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:31:51.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:51.539 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:51.529+0000 7fcd9beb7a40 -1 Falling back to public interface 2026-02-22T20:31:52.141 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:52.129+0000 7fcd9beb7a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:31:52.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:52.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:52.446 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:31:52.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:52.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:52.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:31:52.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:53.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:53.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:53.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:31:53.605 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:31:53.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:53.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:31:53.764 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3839431821,v1:127.0.0.1:6811/3839431821] [v2:127.0.0.1:6812/3839431821,v1:127.0.0.1:6813/3839431821] exists,up c1dca8fe-ebc0-49e3-b0ec-1147c504e603 2026-02-22T20:31:53.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:31:53.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:31:53.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:31:53.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:31:53.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2026-02-22T20:31:53.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:53.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:31:53.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:31:53.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:31:53.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:31:53.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:53.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:53.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:53.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:53.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:31:53.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:31:53.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:53.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:31:53.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:53.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:31:53.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:53.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:53.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:53.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:53.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:31:53.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:31:53.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:31:53.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:31:53.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b8ce7719-88a1-4f18-a1c5-1364d72a09a4 2026-02-22T20:31:53.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 b8ce7719-88a1-4f18-a1c5-1364d72a09a4' 2026-02-22T20:31:53.772 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 b8ce7719-88a1-4f18-a1c5-1364d72a09a4 2026-02-22T20:31:53.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:31:53.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC5Z5tpKNAkLhAAjvpCinhxYEJottyP/jELPg== 2026-02-22T20:31:53.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC5Z5tpKNAkLhAAjvpCinhxYEJottyP/jELPg=="}' 2026-02-22T20:31:53.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b8ce7719-88a1-4f18-a1c5-1364d72a09a4 -i td/osd-scrub-repair/2/new.json 2026-02-22T20:31:53.943 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:31:53.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:31:53.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5Z5tpKNAkLhAAjvpCinhxYEJottyP/jELPg== --osd-uuid b8ce7719-88a1-4f18-a1c5-1364d72a09a4 2026-02-22T20:31:53.980 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:53.969+0000 7f1f946f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:53.982 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:53.973+0000 7f1f946f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:53.983 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:53.973+0000 7f1f946f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:53.983 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:53.973+0000 7f1f946f9a40 -1 bdev(0x5567a003ac00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:31:53.984 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:53.973+0000 7f1f946f9a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:31:54.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:31:54.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:31:54.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:31:54.579 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:31:54.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:31:54.788 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:31:54.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:31:54.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:31:54.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:31:54.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:31:54.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:31:54.812 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:54.801+0000 7f7256d3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:54.813 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:54.801+0000 7f7256d3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:54.815 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:54.805+0000 7f7256d3ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:54.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:31:54.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:54.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:31:54.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:54.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:54.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:54.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:54.945 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:31:54.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:54.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:31:55.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:55.183 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:55.173+0000 7f7256d3ca40 -1 Falling back to public interface 2026-02-22T20:31:55.722 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:55.709+0000 7f7256d3ca40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:31:56.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:56.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:56.107 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:31:56.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:56.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:56.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:31:56.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:57.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:57.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:57.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:31:57.264 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:31:57.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:57.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:31:57.424 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/292049777,v1:127.0.0.1:6819/292049777] [v2:127.0.0.1:6820/292049777,v1:127.0.0.1:6821/292049777] exists,up b8ce7719-88a1-4f18-a1c5-1364d72a09a4 2026-02-22T20:31:57.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:31:57.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:31:57.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:31:57.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:31:57.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2026-02-22T20:31:57.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:31:57.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:31:57.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-22T20:31:57.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:31:57.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:31:57.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:31:57.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:31:57.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:31:57.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:31:57.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:31:57.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:31:57.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:31:57.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:31:57.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:31:57.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:31:57.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:31:57.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:31:57.429 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:31:57.429 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:31:57.429 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:31:57.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:57.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:31:57.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:31:57.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:31:57.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:31:57.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:31:57.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:31:57.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:31:57.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:31:57.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:31:57.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:31:57.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:31:57.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:31:57.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:31:57.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:31:57.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=13ed2ee7-2de8-444f-8bda-cc72d609f742 2026-02-22T20:31:57.432 INFO:tasks.workunit.client.0.trial150.stdout:add osd3 13ed2ee7-2de8-444f-8bda-cc72d609f742 2026-02-22T20:31:57.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 13ed2ee7-2de8-444f-8bda-cc72d609f742' 2026-02-22T20:31:57.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:31:57.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC9Z5tpZ6/eGRAAmljK7NXcw0vgSNrHn5CO+g== 2026-02-22T20:31:57.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC9Z5tpZ6/eGRAAmljK7NXcw0vgSNrHn5CO+g=="}' 2026-02-22T20:31:57.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 13ed2ee7-2de8-444f-8bda-cc72d609f742 -i td/osd-scrub-repair/3/new.json 2026-02-22T20:31:57.605 INFO:tasks.workunit.client.0.trial150.stdout:3 2026-02-22T20:31:57.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-22T20:31:57.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9Z5tpZ6/eGRAAmljK7NXcw0vgSNrHn5CO+g== --osd-uuid 13ed2ee7-2de8-444f-8bda-cc72d609f742 2026-02-22T20:31:57.640 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:57.629+0000 7f9a25e3aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:57.642 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:57.633+0000 7f9a25e3aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:57.643 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:57.633+0000 7f9a25e3aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:57.644 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:57.633+0000 7f9a25e3aa40 -1 bdev(0x557dd3f76c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-22T20:31:57.644 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:57.633+0000 7f9a25e3aa40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-22T20:31:58.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-22T20:31:58.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:31:58.273 INFO:tasks.workunit.client.0.trial150.stdout:adding osd3 key to auth repository 2026-02-22T20:31:58.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-22T20:31:58.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:31:58.474 INFO:tasks.workunit.client.0.trial150.stdout:start osd.3 2026-02-22T20:31:58.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-22T20:31:58.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:31:58.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:31:58.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:31:58.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:31:58.499 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:58.489+0000 7fed9f5a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:58.500 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:58.489+0000 7fed9f5a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:58.501 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:58.489+0000 7fed9f5a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:31:58.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-22T20:31:58.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:31:58.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:31:58.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:31:58.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:31:58.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:58.636 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:31:58.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:31:58.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:58.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:31:58.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:31:58.927 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:58.917+0000 7fed9f5a5a40 -1 Falling back to public interface 2026-02-22T20:31:59.508 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:31:59.497+0000 7fed9f5a5a40 -1 osd.3 0 log_to_monitors true 2026-02-22T20:31:59.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:31:59.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:31:59.795 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:31:59.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:31:59.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:31:59.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:31:59.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:00.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:00.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:00.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:32:00.958 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:32:00.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:00.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:32:01.114 INFO:tasks.workunit.client.0.trial150.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4199356409,v1:127.0.0.1:6827/4199356409] [v2:127.0.0.1:6828/4199356409,v1:127.0.0.1:6829/4199356409] exists,up 13ed2ee7-2de8-444f-8bda-cc72d609f742 2026-02-22T20:32:01.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:32:01.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:32:01.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:32:01.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:32:01.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2026-02-22T20:32:01.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:32:01.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:32:01.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-22T20:32:01.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:32:01.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2026-02-22T20:32:01.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:32:01.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:32:01.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:32:01.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:32:01.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2026-02-22T20:32:01.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2026-02-22T20:32:01.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:32:01.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:32:01.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:32:01.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:32:01.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:32:01.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:32:01.120 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:32:01.120 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:32:01.121 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:32:01.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:01.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:01.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:32:01.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:32:01.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:32:01.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:32:01.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:32:01.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:32:01.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:32:01.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:32:01.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:32:01.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:32:01.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:32:01.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2026-02-22T20:32:01.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:32:01.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f9ba2052-c49c-4f06-88ec-5fcaf52348a3 2026-02-22T20:32:01.123 INFO:tasks.workunit.client.0.trial150.stdout:add osd4 f9ba2052-c49c-4f06-88ec-5fcaf52348a3 2026-02-22T20:32:01.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 f9ba2052-c49c-4f06-88ec-5fcaf52348a3' 2026-02-22T20:32:01.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:32:01.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDBZ5tpa8l2BxAAOOYaxWEFRQnNAdyommlWWg== 2026-02-22T20:32:01.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDBZ5tpa8l2BxAAOOYaxWEFRQnNAdyommlWWg=="}' 2026-02-22T20:32:01.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f9ba2052-c49c-4f06-88ec-5fcaf52348a3 -i td/osd-scrub-repair/4/new.json 2026-02-22T20:32:01.291 INFO:tasks.workunit.client.0.trial150.stdout:4 2026-02-22T20:32:01.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2026-02-22T20:32:01.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBZ5tpa8l2BxAAOOYaxWEFRQnNAdyommlWWg== --osd-uuid f9ba2052-c49c-4f06-88ec-5fcaf52348a3 2026-02-22T20:32:01.319 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:01.309+0000 7fb38b2e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:01.320 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:01.309+0000 7fb38b2e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:01.321 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:01.309+0000 7fb38b2e2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:01.322 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:01.309+0000 7fb38b2e2a40 -1 bdev(0x5561a5512c00 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2026-02-22T20:32:01.322 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:01.309+0000 7fb38b2e2a40 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2026-02-22T20:32:01.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2026-02-22T20:32:01.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:32:01.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-22T20:32:01.950 INFO:tasks.workunit.client.0.trial150.stdout:adding osd4 key to auth repository 2026-02-22T20:32:01.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:32:02.151 INFO:tasks.workunit.client.0.trial150.stdout:start osd.4 2026-02-22T20:32:02.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-22T20:32:02.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:32:02.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:32:02.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:32:02.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:32:02.167 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:02.157+0000 7fd8b0d98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:02.169 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:02.157+0000 7fd8b0d98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:02.170 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:02.161+0000 7fd8b0d98a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:02.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-22T20:32:02.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:32:02.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-22T20:32:02.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:32:02.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:32:02.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:02.315 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:32:02.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:32:02.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:02.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-22T20:32:02.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:02.563 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:02.553+0000 7fd8b0d98a40 -1 Falling back to public interface 2026-02-22T20:32:03.227 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:03.217+0000 7fd8b0d98a40 -1 osd.4 0 log_to_monitors true 2026-02-22T20:32:03.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:03.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:03.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:32:03.481 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:32:03.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:03.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-22T20:32:03.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:04.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:04.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:04.640 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:32:04.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:32:04.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:04.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-22T20:32:04.798 INFO:tasks.workunit.client.0.trial150.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3309903038,v1:127.0.0.1:6835/3309903038] [v2:127.0.0.1:6836/3309903038,v1:127.0.0.1:6837/3309903038] exists,up f9ba2052-c49c-4f06-88ec-5fcaf52348a3 2026-02-22T20:32:04.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:32:04.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:32:04.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:32:04.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:32:04.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2026-02-22T20:32:04.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:32:04.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:32:04.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-22T20:32:04.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:32:04.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:32:04.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:32:04.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:32:04.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:32:04.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:04.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2026-02-22T20:32:04.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:32:04.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b815c525-94b6-4f0c-bea0-1363d5e04434 2026-02-22T20:32:04.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 b815c525-94b6-4f0c-bea0-1363d5e04434' 2026-02-22T20:32:04.807 INFO:tasks.workunit.client.0.trial150.stdout:add osd5 b815c525-94b6-4f0c-bea0-1363d5e04434 2026-02-22T20:32:04.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:32:04.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDEZ5tpsRg3MBAAYmVBD2d2/LB+2Xc9Qgaa7g== 2026-02-22T20:32:04.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDEZ5tpsRg3MBAAYmVBD2d2/LB+2Xc9Qgaa7g=="}' 2026-02-22T20:32:04.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b815c525-94b6-4f0c-bea0-1363d5e04434 -i td/osd-scrub-repair/5/new.json 2026-02-22T20:32:04.976 INFO:tasks.workunit.client.0.trial150.stdout:5 2026-02-22T20:32:04.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2026-02-22T20:32:04.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDEZ5tpsRg3MBAAYmVBD2d2/LB+2Xc9Qgaa7g== --osd-uuid b815c525-94b6-4f0c-bea0-1363d5e04434 2026-02-22T20:32:05.012 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:05.001+0000 7fed9d143a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:05.013 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:05.001+0000 7fed9d143a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:05.015 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:05.005+0000 7fed9d143a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:05.015 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:05.005+0000 7fed9d143a40 -1 bdev(0x5632f6222c00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2026-02-22T20:32:05.015 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:05.005+0000 7fed9d143a40 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2026-02-22T20:32:05.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2026-02-22T20:32:05.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:32:05.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-22T20:32:05.614 INFO:tasks.workunit.client.0.trial150.stdout:adding osd5 key to auth repository 2026-02-22T20:32:05.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:32:05.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-22T20:32:05.811 INFO:tasks.workunit.client.0.trial150.stdout:start osd.5 2026-02-22T20:32:05.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:32:05.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:32:05.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:32:05.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:32:05.834 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:05.825+0000 7f7605210a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:05.836 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:05.825+0000 7f7605210a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:05.837 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:05.829+0000 7f7605210a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:05.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-22T20:32:05.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:32:05.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-22T20:32:05.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:32:05.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:32:05.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:05.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:32:05.980 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:32:05.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:05.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:32:06.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:06.235 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:06.225+0000 7f7605210a40 -1 Falling back to public interface 2026-02-22T20:32:06.924 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:06.913+0000 7f7605210a40 -1 osd.5 0 log_to_monitors true 2026-02-22T20:32:07.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:07.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:07.136 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:32:07.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:32:07.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:07.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:32:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:08.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:08.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:08.297 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:32:08.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:32:08.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:08.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:32:08.459 INFO:tasks.workunit.client.0.trial150.stdout:osd.5 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1942274909,v1:127.0.0.1:6843/1942274909] [v2:127.0.0.1:6844/1942274909,v1:127.0.0.1:6845/1942274909] exists,up b815c525-94b6-4f0c-bea0-1363d5e04434 2026-02-22T20:32:08.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:32:08.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:32:08.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:32:08.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:32:08.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2026-02-22T20:32:08.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:32:08.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:32:08.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-22T20:32:08.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:32:08.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2026-02-22T20:32:08.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:32:08.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:32:08.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:32:08.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:32:08.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2026-02-22T20:32:08.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2026-02-22T20:32:08.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:32:08.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:32:08.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:32:08.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:32:08.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:32:08.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:32:08.464 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:32:08.464 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:32:08.464 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:32:08.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:08.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:32:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:32:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:32:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:32:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:32:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:32:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:32:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:32:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:32:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:32:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:32:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2026-02-22T20:32:08.466 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:32:08.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=25ef1432-50b8-43df-8304-150638788bab 2026-02-22T20:32:08.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 25ef1432-50b8-43df-8304-150638788bab' 2026-02-22T20:32:08.467 INFO:tasks.workunit.client.0.trial150.stdout:add osd6 25ef1432-50b8-43df-8304-150638788bab 2026-02-22T20:32:08.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:32:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDIZ5tpA+z2GxAAswHaFPZMOH9mbO81akvd6A== 2026-02-22T20:32:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDIZ5tpA+z2GxAAswHaFPZMOH9mbO81akvd6A=="}' 2026-02-22T20:32:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 25ef1432-50b8-43df-8304-150638788bab -i td/osd-scrub-repair/6/new.json 2026-02-22T20:32:08.639 INFO:tasks.workunit.client.0.trial150.stdout:6 2026-02-22T20:32:08.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2026-02-22T20:32:08.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDIZ5tpA+z2GxAAswHaFPZMOH9mbO81akvd6A== --osd-uuid 25ef1432-50b8-43df-8304-150638788bab 2026-02-22T20:32:08.664 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:08.653+0000 7f0e60b90a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:08.666 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:08.657+0000 7f0e60b90a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:08.667 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:08.657+0000 7f0e60b90a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:08.667 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:08.657+0000 7f0e60b90a40 -1 bdev(0x55724b1bac00 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2026-02-22T20:32:08.667 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:08.657+0000 7f0e60b90a40 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2026-02-22T20:32:09.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2026-02-22T20:32:09.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:32:09.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-22T20:32:09.302 INFO:tasks.workunit.client.0.trial150.stdout:adding osd6 key to auth repository 2026-02-22T20:32:09.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:32:09.504 INFO:tasks.workunit.client.0.trial150.stdout:start osd.6 2026-02-22T20:32:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-22T20:32:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:32:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:32:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:32:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:32:09.528 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:09.517+0000 7f612d147a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:09.529 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:09.517+0000 7f612d147a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:09.530 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:09.521+0000 7f612d147a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:09.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-22T20:32:09.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:32:09.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-22T20:32:09.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:32:09.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:32:09.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:09.664 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:32:09.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:32:09.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:09.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-22T20:32:09.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:09.903 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:09.893+0000 7f612d147a40 -1 Falling back to public interface 2026-02-22T20:32:10.460 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:10.449+0000 7f612d147a40 -1 osd.6 0 log_to_monitors true 2026-02-22T20:32:10.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:10.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:10.821 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:32:10.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:32:10.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:10.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-22T20:32:10.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:11.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:11.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:11.994 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:32:11.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:32:11.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:11.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-22T20:32:12.146 INFO:tasks.workunit.client.0.trial150.stdout:osd.6 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/3678837444,v1:127.0.0.1:6851/3678837444] [v2:127.0.0.1:6852/3678837444,v1:127.0.0.1:6853/3678837444] exists,up 25ef1432-50b8-43df-8304-150638788bab 2026-02-22T20:32:12.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:32:12.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:32:12.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:32:12.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:32:12.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2026-02-22T20:32:12.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:32:12.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:32:12.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-22T20:32:12.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:32:12.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2026-02-22T20:32:12.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:32:12.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:32:12.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:32:12.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:32:12.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2026-02-22T20:32:12.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2026-02-22T20:32:12.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:32:12.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:32:12.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:32:12.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:32:12.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:32:12.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:32:12.151 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:32:12.151 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:32:12.151 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:32:12.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:32:12.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2026-02-22T20:32:12.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:32:12.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7a73809e-4a10-441f-9ff1-044f3e316ade 2026-02-22T20:32:12.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 7a73809e-4a10-441f-9ff1-044f3e316ade' 2026-02-22T20:32:12.154 INFO:tasks.workunit.client.0.trial150.stdout:add osd7 7a73809e-4a10-441f-9ff1-044f3e316ade 2026-02-22T20:32:12.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:32:12.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDMZ5tplPiECRAAkRvzGAtvQifa0zOyssPG9A== 2026-02-22T20:32:12.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDMZ5tplPiECRAAkRvzGAtvQifa0zOyssPG9A=="}' 2026-02-22T20:32:12.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7a73809e-4a10-441f-9ff1-044f3e316ade -i td/osd-scrub-repair/7/new.json 2026-02-22T20:32:12.319 INFO:tasks.workunit.client.0.trial150.stdout:7 2026-02-22T20:32:12.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2026-02-22T20:32:12.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMZ5tplPiECRAAkRvzGAtvQifa0zOyssPG9A== --osd-uuid 7a73809e-4a10-441f-9ff1-044f3e316ade 2026-02-22T20:32:12.346 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:12.337+0000 7f263f354a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:12.348 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:12.337+0000 7f263f354a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:12.349 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:12.337+0000 7f263f354a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:12.350 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:12.337+0000 7f263f354a40 -1 bdev(0x560a9daccc00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2026-02-22T20:32:12.350 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:12.337+0000 7f263f354a40 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2026-02-22T20:32:13.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2026-02-22T20:32:13.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:32:13.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-22T20:32:13.030 INFO:tasks.workunit.client.0.trial150.stdout:adding osd7 key to auth repository 2026-02-22T20:32:13.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:32:13.224 INFO:tasks.workunit.client.0.trial150.stdout:start osd.7 2026-02-22T20:32:13.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-22T20:32:13.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:32:13.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:32:13.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:32:13.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:32:13.240 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:13.229+0000 7f912f2cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:13.242 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:13.233+0000 7f912f2cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:13.243 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:13.233+0000 7f912f2cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:13.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-22T20:32:13.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:32:13.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-22T20:32:13.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:32:13.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:32:13.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:13.386 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:32:13.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:32:13.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:13.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-22T20:32:13.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:13.639 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:13.629+0000 7f912f2cba40 -1 Falling back to public interface 2026-02-22T20:32:14.234 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:14.221+0000 7f912f2cba40 -1 osd.7 0 log_to_monitors true 2026-02-22T20:32:14.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:14.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:14.546 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:32:14.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:32:14.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:14.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-22T20:32:14.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:15.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:15.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:15.708 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:32:15.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:32:15.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:15.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-22T20:32:15.868 INFO:tasks.workunit.client.0.trial150.stdout:osd.7 up in weight 1 up_from 37 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/497277663,v1:127.0.0.1:6859/497277663] [v2:127.0.0.1:6860/497277663,v1:127.0.0.1:6861/497277663] exists,up 7a73809e-4a10-441f-9ff1-044f3e316ade 2026-02-22T20:32:15.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:32:15.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:32:15.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:32:15.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:32:15.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2026-02-22T20:32:15.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:32:15.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:32:15.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-22T20:32:15.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:32:15.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2026-02-22T20:32:15.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:32:15.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:32:15.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:32:15.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:32:15.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2026-02-22T20:32:15.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2026-02-22T20:32:15.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:32:15.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:32:15.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:32:15.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:32:15.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:32:15.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:32:15.874 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:32:15.874 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:32:15.874 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:32:15.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:15.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:15.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:32:15.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:32:15.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:32:15.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:32:15.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:32:15.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:32:15.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:32:15.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:32:15.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:32:15.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:32:15.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:32:15.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2026-02-22T20:32:15.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:32:15.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2a2ae659-04bc-4279-8d9d-34e5ecd2719b 2026-02-22T20:32:15.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 2a2ae659-04bc-4279-8d9d-34e5ecd2719b' 2026-02-22T20:32:15.877 INFO:tasks.workunit.client.0.trial150.stdout:add osd8 2a2ae659-04bc-4279-8d9d-34e5ecd2719b 2026-02-22T20:32:15.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:32:15.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDPZ5tplelTNBAAQHUYkaWu4vYYS9PBOcGpSg== 2026-02-22T20:32:15.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDPZ5tplelTNBAAQHUYkaWu4vYYS9PBOcGpSg=="}' 2026-02-22T20:32:15.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2a2ae659-04bc-4279-8d9d-34e5ecd2719b -i td/osd-scrub-repair/8/new.json 2026-02-22T20:32:16.044 INFO:tasks.workunit.client.0.trial150.stdout:8 2026-02-22T20:32:16.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2026-02-22T20:32:16.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPZ5tplelTNBAAQHUYkaWu4vYYS9PBOcGpSg== --osd-uuid 2a2ae659-04bc-4279-8d9d-34e5ecd2719b 2026-02-22T20:32:16.071 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:16.061+0000 7f85855b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:16.073 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:16.061+0000 7f85855b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:16.074 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:16.061+0000 7f85855b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:16.074 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:16.065+0000 7f85855b7a40 -1 bdev(0x55df86452c00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2026-02-22T20:32:16.074 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:16.065+0000 7f85855b7a40 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2026-02-22T20:32:16.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2026-02-22T20:32:16.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:32:16.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-22T20:32:16.731 INFO:tasks.workunit.client.0.trial150.stdout:adding osd8 key to auth repository 2026-02-22T20:32:16.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:32:16.931 INFO:tasks.workunit.client.0.trial150.stdout:start osd.8 2026-02-22T20:32:16.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-22T20:32:16.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:32:16.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:32:16.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:32:16.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:32:16.957 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:16.945+0000 7f1fd3d41a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:16.958 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:16.949+0000 7f1fd3d41a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:16.959 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:16.949+0000 7f1fd3d41a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:17.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-22T20:32:17.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:32:17.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-22T20:32:17.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:32:17.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:32:17.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:17.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:32:17.093 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:32:17.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:17.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-22T20:32:17.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:17.308 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:17.297+0000 7f1fd3d41a40 -1 Falling back to public interface 2026-02-22T20:32:17.906 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:17.893+0000 7f1fd3d41a40 -1 osd.8 0 log_to_monitors true 2026-02-22T20:32:18.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:18.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:18.255 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:32:18.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:32:18.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:18.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-22T20:32:18.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:19.413 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:32:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:32:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-22T20:32:19.568 INFO:tasks.workunit.client.0.trial150.stdout:osd.8 up in weight 1 up_from 42 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/4093037981,v1:127.0.0.1:6867/4093037981] [v2:127.0.0.1:6868/4093037981,v1:127.0.0.1:6869/4093037981] exists,up 2a2ae659-04bc-4279-8d9d-34e5ecd2719b 2026-02-22T20:32:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:32:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:32:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:32:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:32:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2026-02-22T20:32:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:32:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:32:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2026-02-22T20:32:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:32:19.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:32:19.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:32:19.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:32:19.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:32:19.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:32:19.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-22T20:32:19.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-22T20:32:19.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:32:19.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:32:19.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:32:19.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:32:19.574 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:32:19.574 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:32:19.574 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:32:19.574 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:32:19.574 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:32:19.574 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:19.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:19.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:32:19.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:32:19.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:32:19.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:32:19.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:32:19.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:32:19.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:32:19.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:32:19.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:32:19.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:32:19.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:32:19.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2026-02-22T20:32:19.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:32:19.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0871896b-7ed0-47d4-99e7-9d74df750764 2026-02-22T20:32:19.576 INFO:tasks.workunit.client.0.trial150.stdout:add osd9 0871896b-7ed0-47d4-99e7-9d74df750764 2026-02-22T20:32:19.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 0871896b-7ed0-47d4-99e7-9d74df750764' 2026-02-22T20:32:19.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:32:19.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDTZ5tpXqpxIhAAW2b69RBGMURBwnpEgJoG2A== 2026-02-22T20:32:19.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDTZ5tpXqpxIhAAW2b69RBGMURBwnpEgJoG2A=="}' 2026-02-22T20:32:19.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0871896b-7ed0-47d4-99e7-9d74df750764 -i td/osd-scrub-repair/9/new.json 2026-02-22T20:32:19.747 INFO:tasks.workunit.client.0.trial150.stdout:9 2026-02-22T20:32:19.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2026-02-22T20:32:19.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDTZ5tpXqpxIhAAW2b69RBGMURBwnpEgJoG2A== --osd-uuid 0871896b-7ed0-47d4-99e7-9d74df750764 2026-02-22T20:32:19.773 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:19.761+0000 7f1c7166ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:19.774 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:19.765+0000 7f1c7166ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:19.775 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:19.765+0000 7f1c7166ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:19.776 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:19.765+0000 7f1c7166ca40 -1 bdev(0x55d721ae6c00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2026-02-22T20:32:19.776 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:19.765+0000 7f1c7166ca40 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2026-02-22T20:32:20.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2026-02-22T20:32:20.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:32:20.422 INFO:tasks.workunit.client.0.trial150.stdout:adding osd9 key to auth repository 2026-02-22T20:32:20.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2026-02-22T20:32:20.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:32:20.611 INFO:tasks.workunit.client.0.trial150.stdout:start osd.9 2026-02-22T20:32:20.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2026-02-22T20:32:20.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:32:20.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:32:20.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:32:20.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:32:20.635 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:20.625+0000 7f2fdb293a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:20.637 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:20.625+0000 7f2fdb293a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:20.638 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:20.629+0000 7f2fdb293a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:20.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2026-02-22T20:32:20.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:32:20.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-22T20:32:20.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:32:20.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:32:20.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:20.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:32:20.774 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:32:20.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:20.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:32:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:20.980 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:20.969+0000 7f2fdb293a40 -1 Falling back to public interface 2026-02-22T20:32:21.639 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:21.629+0000 7f2fdb293a40 -1 osd.9 0 log_to_monitors true 2026-02-22T20:32:21.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:21.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:21.939 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:32:21.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:32:21.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:21.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:32:22.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:23.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:23.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:23.097 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:32:23.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:32:23.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:23.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:32:23.257 INFO:tasks.workunit.client.0.trial150.stdout:osd.9 up in weight 1 up_from 46 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/812025676,v1:127.0.0.1:6875/812025676] [v2:127.0.0.1:6876/812025676,v1:127.0.0.1:6877/812025676] exists,up 0871896b-7ed0-47d4-99e7-9d74df750764 2026-02-22T20:32:23.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:32:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:32:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:32:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: corrupt_and_repair_lrc: create_rbd_pool 2026-02-22T20:32:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:32:23.405 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:32:23.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:32:23.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:32:23.600 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:32:23.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:32:24.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:32:24.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: corrupt_and_repair_lrc: wait_for_clean 2026-02-22T20:32:24.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:32:24.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:32:24.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:32:24.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:32:24.859 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:32:24.860 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:32:24.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:32:24.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:32:24.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:32:24.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:32:24.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:32:24.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:32:24.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:32:24.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:32:24.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:32:25.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:32:25.133 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:32:25.133 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:32:25.133 INFO:tasks.workunit.client.0.trial150.stderr:3 2026-02-22T20:32:25.133 INFO:tasks.workunit.client.0.trial150.stderr:4 2026-02-22T20:32:25.133 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:32:25.133 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:32:25.134 INFO:tasks.workunit.client.0.trial150.stderr:7 2026-02-22T20:32:25.134 INFO:tasks.workunit.client.0.trial150.stderr:8 2026-02-22T20:32:25.134 INFO:tasks.workunit.client.0.trial150.stderr:9' 2026-02-22T20:32:25.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:32:25.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:25.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:32:25.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-22T20:32:25.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-22T20:32:25.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-22T20:32:25.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:25.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:32:25.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672968 2026-02-22T20:32:25.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672968 2026-02-22T20:32:25.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2026-02-22T20:32:25.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:25.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:32:25.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542151 2026-02-22T20:32:25.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542151 2026-02-22T20:32:25.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-60129542151' 2026-02-22T20:32:25.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:25.349 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:32:25.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378630 2026-02-22T20:32:25.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378630 2026-02-22T20:32:25.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-60129542151 3-81604378630' 2026-02-22T20:32:25.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:25.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:32:25.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215110 2026-02-22T20:32:25.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215110 2026-02-22T20:32:25.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-60129542151 3-81604378630 4-103079215110' 2026-02-22T20:32:25.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:25.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:32:25.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084293 2026-02-22T20:32:25.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084293 2026-02-22T20:32:25.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-60129542151 3-81604378630 4-103079215110 5-120259084293' 2026-02-22T20:32:25.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:25.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:32:25.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953476 2026-02-22T20:32:25.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953476 2026-02-22T20:32:25.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-60129542151 3-81604378630 4-103079215110 5-120259084293 6-137438953476' 2026-02-22T20:32:25.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:25.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:32:25.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789956 2026-02-22T20:32:25.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789956 2026-02-22T20:32:25.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-60129542151 3-81604378630 4-103079215110 5-120259084293 6-137438953476 7-158913789956' 2026-02-22T20:32:25.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:25.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:32:25.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626435 2026-02-22T20:32:25.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626435 2026-02-22T20:32:25.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-60129542151 3-81604378630 4-103079215110 5-120259084293 6-137438953476 7-158913789956 8-180388626435' 2026-02-22T20:32:25.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:25.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:32:25.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=197568495618 2026-02-22T20:32:25.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 197568495618 2026-02-22T20:32:25.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-60129542151 3-81604378630 4-103079215110 5-120259084293 6-137438953476 7-158913789956 8-180388626435 9-197568495618' 2026-02-22T20:32:25.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:25.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-22T20:32:25.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:25.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:32:25.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-22T20:32:25.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:25.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-22T20:32:25.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-22T20:32:25.861 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836489 2026-02-22T20:32:25.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:32:26.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836489 2026-02-22T20:32:26.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:32:27.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:32:27.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:32:27.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-22T20:32:27.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:27.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672968 2026-02-22T20:32:27.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:27.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:32:27.184 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672968 2026-02-22T20:32:27.184 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:27.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672968 2026-02-22T20:32:27.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2026-02-22T20:32:27.186 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672968 2026-02-22T20:32:27.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:32:27.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672968 2026-02-22T20:32:27.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:27.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542151 2026-02-22T20:32:27.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:27.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:32:27.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542151 2026-02-22T20:32:27.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:27.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542151 2026-02-22T20:32:27.362 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542151 2026-02-22T20:32:27.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2026-02-22T20:32:27.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:32:27.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542151 -lt 60129542151 2026-02-22T20:32:27.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:27.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378630 2026-02-22T20:32:27.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:27.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:32:27.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-81604378630 2026-02-22T20:32:27.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:27.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=81604378630 2026-02-22T20:32:27.533 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.3 seq 81604378630 2026-02-22T20:32:27.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 81604378630' 2026-02-22T20:32:27.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:32:27.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 81604378631 -lt 81604378630 2026-02-22T20:32:27.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:27.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215110 2026-02-22T20:32:27.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:27.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:32:27.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215110 2026-02-22T20:32:27.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:27.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215110 2026-02-22T20:32:27.698 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.4 seq 103079215110 2026-02-22T20:32:27.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215110' 2026-02-22T20:32:27.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:32:27.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215110 -lt 103079215110 2026-02-22T20:32:27.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:27.867 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-120259084293 2026-02-22T20:32:27.867 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:27.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:32:27.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-120259084293 2026-02-22T20:32:27.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:27.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084293 2026-02-22T20:32:27.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 120259084293' 2026-02-22T20:32:27.870 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.5 seq 120259084293 2026-02-22T20:32:27.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:32:28.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084293 -lt 120259084293 2026-02-22T20:32:28.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:28.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953476 2026-02-22T20:32:28.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:28.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:32:28.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953476 2026-02-22T20:32:28.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:28.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953476 2026-02-22T20:32:28.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953476' 2026-02-22T20:32:28.045 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.6 seq 137438953476 2026-02-22T20:32:28.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:32:28.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953477 -lt 137438953476 2026-02-22T20:32:28.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:28.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789956 2026-02-22T20:32:28.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:28.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:32:28.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789956 2026-02-22T20:32:28.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:28.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789956 2026-02-22T20:32:28.209 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.7 seq 158913789956 2026-02-22T20:32:28.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789956' 2026-02-22T20:32:28.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:32:28.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789956 -lt 158913789956 2026-02-22T20:32:28.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:28.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626435 2026-02-22T20:32:28.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:28.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:32:28.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626435 2026-02-22T20:32:28.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:28.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626435 2026-02-22T20:32:28.381 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.8 seq 180388626435 2026-02-22T20:32:28.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626435' 2026-02-22T20:32:28.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:32:28.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626435 -lt 180388626435 2026-02-22T20:32:28.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:28.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-197568495618 2026-02-22T20:32:28.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:28.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:32:28.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-197568495618 2026-02-22T20:32:28.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:28.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=197568495618 2026-02-22T20:32:28.546 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.9 seq 197568495618 2026-02-22T20:32:28.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 197568495618' 2026-02-22T20:32:28.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:32:28.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 197568495618 -lt 197568495618 2026-02-22T20:32:28.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:32:28.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:32:28.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:32:28.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:32:28.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:32:28.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:32:28.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:32:28.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:32:28.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:32:28.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:32:28.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:32:29.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:32:29.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:32:29.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:32:29.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:32:29.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:32:29.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:32:29.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:32:29.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:896: corrupt_and_repair_lrc: create_ec_pool ecpool false k=4 m=2 l=3 plugin=lrc 2026-02-22T20:32:29.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-22T20:32:29.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:32:29.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-22T20:32:29.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:32:29.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2026-02-22T20:32:29.502 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:32:29.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-22T20:32:29.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-22T20:32:29.807 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' created 2026-02-22T20:32:29.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:32:30.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-22T20:32:30.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:32:30.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:32:30.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:32:30.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:32:30.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:32:30.820 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:32:30.820 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:32:30.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:32:30.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:32:30.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:32:30.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:32:30.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:32:30.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:32:30.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:32:30.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:32:30.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr:3 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr:4 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr:7 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr:8 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr:9' 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:31.090 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:32:31.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-22T20:32:31.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-22T20:32:31.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T20:32:31.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:31.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:32:31.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672970 2026-02-22T20:32:31.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672970 2026-02-22T20:32:31.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2026-02-22T20:32:31.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:31.233 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:32:31.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-22T20:32:31.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-22T20:32:31.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153' 2026-02-22T20:32:31.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:31.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:32:31.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378633 2026-02-22T20:32:31.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378633 2026-02-22T20:32:31.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153 3-81604378633' 2026-02-22T20:32:31.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:31.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:32:31.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215112 2026-02-22T20:32:31.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215112 2026-02-22T20:32:31.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153 3-81604378633 4-103079215112' 2026-02-22T20:32:31.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:31.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:32:31.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084295 2026-02-22T20:32:31.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084295 2026-02-22T20:32:31.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153 3-81604378633 4-103079215112 5-120259084295' 2026-02-22T20:32:31.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:31.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:32:31.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953478 2026-02-22T20:32:31.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953478 2026-02-22T20:32:31.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153 3-81604378633 4-103079215112 5-120259084295 6-137438953478' 2026-02-22T20:32:31.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:31.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:32:31.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789958 2026-02-22T20:32:31.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789958 2026-02-22T20:32:31.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153 3-81604378633 4-103079215112 5-120259084295 6-137438953478 7-158913789958' 2026-02-22T20:32:31.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:31.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:32:31.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626437 2026-02-22T20:32:31.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626437 2026-02-22T20:32:31.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153 3-81604378633 4-103079215112 5-120259084295 6-137438953478 7-158913789958 8-180388626437' 2026-02-22T20:32:31.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:31.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:32:31.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=197568495620 2026-02-22T20:32:31.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 197568495620 2026-02-22T20:32:31.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153 3-81604378633 4-103079215112 5-120259084295 6-137438953478 7-158913789958 8-180388626437 9-197568495620' 2026-02-22T20:32:31.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:31.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-22T20:32:31.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:31.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:32:31.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-22T20:32:31.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:31.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-22T20:32:31.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T20:32:31.813 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836491 2026-02-22T20:32:31.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:32:31.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T20:32:31.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:32:32.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:32:32.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:32:33.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-22T20:32:33.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:33.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672970 2026-02-22T20:32:33.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:33.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:32:33.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672970 2026-02-22T20:32:33.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:33.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672970 2026-02-22T20:32:33.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2026-02-22T20:32:33.144 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672970 2026-02-22T20:32:33.145 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:32:33.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672970 -lt 42949672970 2026-02-22T20:32:33.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:33.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-22T20:32:33.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:33.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:32:33.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-22T20:32:33.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:33.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-22T20:32:33.307 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542153 2026-02-22T20:32:33.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-22T20:32:33.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:32:33.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542154 -lt 60129542153 2026-02-22T20:32:33.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:33.472 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378633 2026-02-22T20:32:33.472 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:33.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:32:33.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-81604378633 2026-02-22T20:32:33.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:33.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=81604378633 2026-02-22T20:32:33.476 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.3 seq 81604378633 2026-02-22T20:32:33.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 81604378633' 2026-02-22T20:32:33.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:32:33.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 81604378633 -lt 81604378633 2026-02-22T20:32:33.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:33.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215112 2026-02-22T20:32:33.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:33.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:32:33.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215112 2026-02-22T20:32:33.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215112 2026-02-22T20:32:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215112' 2026-02-22T20:32:33.642 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.4 seq 103079215112 2026-02-22T20:32:33.642 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:32:33.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215112 -lt 103079215112 2026-02-22T20:32:33.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:33.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-120259084295 2026-02-22T20:32:33.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:33.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:32:33.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-120259084295 2026-02-22T20:32:33.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:33.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084295 2026-02-22T20:32:33.807 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.5 seq 120259084295 2026-02-22T20:32:33.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 120259084295' 2026-02-22T20:32:33.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:32:33.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084295 -lt 120259084295 2026-02-22T20:32:33.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:33.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953478 2026-02-22T20:32:33.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:33.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:32:33.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953478 2026-02-22T20:32:33.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:33.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953478 2026-02-22T20:32:33.975 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.6 seq 137438953478 2026-02-22T20:32:33.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953478' 2026-02-22T20:32:33.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:32:34.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953479 -lt 137438953478 2026-02-22T20:32:34.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:34.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789958 2026-02-22T20:32:34.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:34.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:32:34.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789958 2026-02-22T20:32:34.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:34.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789958 2026-02-22T20:32:34.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789958' 2026-02-22T20:32:34.145 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.7 seq 158913789958 2026-02-22T20:32:34.145 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:32:34.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789958 -lt 158913789958 2026-02-22T20:32:34.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:34.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626437 2026-02-22T20:32:34.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:34.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:32:34.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626437 2026-02-22T20:32:34.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:34.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626437 2026-02-22T20:32:34.309 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.8 seq 180388626437 2026-02-22T20:32:34.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626437' 2026-02-22T20:32:34.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:32:34.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626437 -lt 180388626437 2026-02-22T20:32:34.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:34.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-197568495620 2026-02-22T20:32:34.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:34.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:32:34.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-197568495620 2026-02-22T20:32:34.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:34.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=197568495620 2026-02-22T20:32:34.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 197568495620' 2026-02-22T20:32:34.480 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.9 seq 197568495620 2026-02-22T20:32:34.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:32:34.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 197568495621 -lt 197568495620 2026-02-22T20:32:34.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:32:34.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:32:34.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:32:34.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:32:34.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:32:34.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:32:34.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:32:34.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:32:34.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:32:34.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:32:34.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:32:35.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:32:35.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:32:35.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:32:35.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: corrupt_and_repair_erasure_coded: local poolname=ecpool 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:32:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:32:35.397 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:32:35.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:32:35.586 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:32:35.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:32:35.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:32:35.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:32:35.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2026-02-22T20:32:35.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-22T20:32:35.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:32:35.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:32:35.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:32:35.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: local primary=3 2026-02-22T20:32:35.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2026-02-22T20:32:35.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: sed -e s/3// 2026-02-22T20:32:35.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-22T20:32:35.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-22T20:32:35.784 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:32:35.784 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:32:35.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-22T20:32:35.940 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:32:35.940 INFO:tasks.workunit.client.0.trial150.stderr:9 2026-02-22T20:32:35.941 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:32:35.941 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:32:35.941 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:32:35.941 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:32:35.941 INFO:tasks.workunit.client.0.trial150.stderr:7' 2026-02-22T20:32:35.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2026-02-22T20:32:35.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2026-02-22T20:32:35.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: local -a osds 2026-02-22T20:32:35.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: corrupt_and_repair_erasure_coded: local not_primary_first=5 2026-02-22T20:32:35.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: corrupt_and_repair_erasure_coded: local not_primary_second=9 2026-02-22T20:32:35.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2026-02-22T20:32:35.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-22T20:32:35.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-22T20:32:35.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=3 2026-02-22T20:32:35.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:32:35.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:32:35.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:32:35.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:32:35.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:32:35.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:32:35.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:32:35.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:32:35.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:32:35.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:32:35.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:32:35.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:32:35.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:32:35.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:32:35.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:32:35.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:32:36.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:32:36.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:32:36.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:32:36.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:32:36.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:32:36.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:32:36.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:32:36.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-22T20:32:36.454 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:32:36.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:32:36.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:32:36.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:32:36.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:32:36.564 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:32:36.564 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:32:36.564 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:32:36.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:36.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:36.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:32:36.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:32:36.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:32:36.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:32:36.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:32:36.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:32:36.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:32:36.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:32:36.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:32:36.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:32:36.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:32:36.566 INFO:tasks.workunit.client.0.trial150.stderr:start osd.3 2026-02-22T20:32:36.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:32:36.566 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:32:36.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:32:36.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:32:36.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:32:36.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:32:36.585 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:36.573+0000 7fcd33a02a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:36.587 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:36.577+0000 7fcd33a02a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:36.588 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:36.577+0000 7fcd33a02a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:36.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:32:36.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:32:36.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:32:36.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:32:36.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:32:36.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:36.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:32:36.730 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:32:36.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:36.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:32:36.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:36.960 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:36.949+0000 7fcd33a02a40 -1 Falling back to public interface 2026-02-22T20:32:37.560 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:37.549+0000 7fcd33a02a40 -1 osd.3 62 log_to_monitors true 2026-02-22T20:32:37.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:37.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:37.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:32:37.889 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:32:37.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:37.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:32:38.046 INFO:tasks.workunit.client.0.trial150.stderr:osd.3 up in weight 1 up_from 66 up_thru 66 down_at 63 last_clean_interval [19,62) [v2:127.0.0.1:6826/2515428853,v1:127.0.0.1:6827/2515428853] [v2:127.0.0.1:6828/2515428853,v1:127.0.0.1:6829/2515428853] exists,up 13ed2ee7-2de8-444f-8bda-cc72d609f742 2026-02-22T20:32:38.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:32:38.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:32:38.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:32:38.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:32:38.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:32:38.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:32:38.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:32:38.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:32:38.047 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:32:38.047 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:32:38.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:32:38.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:32:38.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:32:38.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:32:38.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:32:38.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:32:38.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:32:38.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:32:38.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr:3 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr:4 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr:7 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr:8 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr:9' 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:38.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:32:38.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-22T20:32:38.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-22T20:32:38.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-22T20:32:38.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:38.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:32:38.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672973 2026-02-22T20:32:38.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672973 2026-02-22T20:32:38.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973' 2026-02-22T20:32:38.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:38.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:32:38.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542156 2026-02-22T20:32:38.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542156 2026-02-22T20:32:38.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-60129542156' 2026-02-22T20:32:38.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:38.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:32:38.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841538 2026-02-22T20:32:38.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841538 2026-02-22T20:32:38.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-60129542156 3-283467841538' 2026-02-22T20:32:38.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:38.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:32:38.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215114 2026-02-22T20:32:38.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215114 2026-02-22T20:32:38.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-60129542156 3-283467841538 4-103079215114' 2026-02-22T20:32:38.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:38.680 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:32:38.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084298 2026-02-22T20:32:38.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084298 2026-02-22T20:32:38.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-60129542156 3-283467841538 4-103079215114 5-120259084298' 2026-02-22T20:32:38.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:38.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:32:38.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953481 2026-02-22T20:32:38.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953481 2026-02-22T20:32:38.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-60129542156 3-283467841538 4-103079215114 5-120259084298 6-137438953481' 2026-02-22T20:32:38.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:38.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:32:38.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789960 2026-02-22T20:32:38.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789960 2026-02-22T20:32:38.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-60129542156 3-283467841538 4-103079215114 5-120259084298 6-137438953481 7-158913789960' 2026-02-22T20:32:38.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:38.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:32:38.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626439 2026-02-22T20:32:38.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626439 2026-02-22T20:32:38.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-60129542156 3-283467841538 4-103079215114 5-120259084298 6-137438953481 7-158913789960 8-180388626439' 2026-02-22T20:32:38.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:38.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:32:39.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=197568495623 2026-02-22T20:32:39.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 197568495623 2026-02-22T20:32:39.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-60129542156 3-283467841538 4-103079215114 5-120259084298 6-137438953481 7-158913789960 8-180388626439 9-197568495623' 2026-02-22T20:32:39.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:39.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-22T20:32:39.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:39.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:32:39.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-22T20:32:39.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:39.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-22T20:32:39.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-22T20:32:39.043 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836493 2026-02-22T20:32:39.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:32:39.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-22T20:32:39.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:39.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672973 2026-02-22T20:32:39.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:39.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:32:39.213 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672973 2026-02-22T20:32:39.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:39.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672973 2026-02-22T20:32:39.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2026-02-22T20:32:39.215 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672973 2026-02-22T20:32:39.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:32:39.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672973 -lt 42949672973 2026-02-22T20:32:39.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:39.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542156 2026-02-22T20:32:39.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:39.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:32:39.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542156 2026-02-22T20:32:39.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:39.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542156 2026-02-22T20:32:39.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2026-02-22T20:32:39.381 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542156 2026-02-22T20:32:39.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:32:39.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542156 -lt 60129542156 2026-02-22T20:32:39.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:39.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841538 2026-02-22T20:32:39.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:39.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:32:39.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841538 2026-02-22T20:32:39.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:39.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841538 2026-02-22T20:32:39.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841538' 2026-02-22T20:32:39.552 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 283467841538 2026-02-22T20:32:39.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:32:39.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841538 -lt 283467841538 2026-02-22T20:32:39.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:39.720 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215114 2026-02-22T20:32:39.720 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:39.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:32:39.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215114 2026-02-22T20:32:39.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:39.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215114 2026-02-22T20:32:39.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215114' 2026-02-22T20:32:39.723 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.4 seq 103079215114 2026-02-22T20:32:39.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:32:39.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215114 -lt 103079215114 2026-02-22T20:32:39.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:39.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-120259084298 2026-02-22T20:32:39.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:39.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:32:39.882 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-120259084298 2026-02-22T20:32:39.882 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:39.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084298 2026-02-22T20:32:39.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 120259084298' 2026-02-22T20:32:39.884 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.5 seq 120259084298 2026-02-22T20:32:39.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:32:40.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084298 -lt 120259084298 2026-02-22T20:32:40.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:40.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953481 2026-02-22T20:32:40.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:40.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:32:40.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953481 2026-02-22T20:32:40.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:40.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953481 2026-02-22T20:32:40.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953481' 2026-02-22T20:32:40.049 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.6 seq 137438953481 2026-02-22T20:32:40.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:32:40.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953480 -lt 137438953481 2026-02-22T20:32:40.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:32:41.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:32:41.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:32:41.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953481 -lt 137438953481 2026-02-22T20:32:41.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:41.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789960 2026-02-22T20:32:41.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:41.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:32:41.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789960 2026-02-22T20:32:41.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:41.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789960 2026-02-22T20:32:41.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789960' 2026-02-22T20:32:41.375 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.7 seq 158913789960 2026-02-22T20:32:41.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:32:41.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789961 -lt 158913789960 2026-02-22T20:32:41.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:41.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626439 2026-02-22T20:32:41.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:41.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:32:41.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626439 2026-02-22T20:32:41.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:41.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626439 2026-02-22T20:32:41.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626439' 2026-02-22T20:32:41.542 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.8 seq 180388626439 2026-02-22T20:32:41.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:32:41.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626440 -lt 180388626439 2026-02-22T20:32:41.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:41.703 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-197568495623 2026-02-22T20:32:41.703 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:41.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:32:41.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-197568495623 2026-02-22T20:32:41.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:41.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=197568495623 2026-02-22T20:32:41.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 197568495623' 2026-02-22T20:32:41.707 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.9 seq 197568495623 2026-02-22T20:32:41.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:32:41.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 197568495623 -lt 197568495623 2026-02-22T20:32:41.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:32:41.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:32:41.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:32:42.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:32:42.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:32:42.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:32:42.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:32:42.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:32:42.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:32:42.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:32:42.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:32:42.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:32:42.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:32:42.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:32:42.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:32:42.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:32:42.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:32:42.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:32:42.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-22T20:32:42.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:32:42.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:32:42.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:32:42.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:32:42.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-22T20:32:42.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-22T20:32:42.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:32:42.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:32:42.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:32:42.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:32:42.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:32:42.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:32:42.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:32:29.798348+0000 2026-02-22T20:32:42.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:32:42.902 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:32:42.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:32:29.798348+0000 2026-02-22T20:32:42.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:32:42.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:32:29.798348+0000 2026-02-22T20:32:42.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:32:42.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:32:42.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:32:42.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:32:42.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:32:42.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:32:42.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:32:42.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:32:43.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:32:29.798348+0000 '>' 2026-02-22T20:32:29.798348+0000 2026-02-22T20:32:43.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:32:44.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:32:44.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:32:44.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:32:44.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:32:44.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:32:44.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:32:44.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:32:44.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:32:29.798348+0000 '>' 2026-02-22T20:32:29.798348+0000 2026-02-22T20:32:44.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:32:45.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:32:45.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:32:45.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:32:45.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:32:45.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:32:45.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:32:45.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:32:45.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:32:29.798348+0000 '>' 2026-02-22T20:32:29.798348+0000 2026-02-22T20:32:45.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:32:46.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:32:46.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:32:46.401 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:32:46.401 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:32:46.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:32:46.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:32:46.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:32:46.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:32:29.798348+0000 '>' 2026-02-22T20:32:29.798348+0000 2026-02-22T20:32:46.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:32:47.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:32:47.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:32:47.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:32:47.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:32:47.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:32:47.566 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:32:47.566 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:32:47.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:32:29.798348+0000 '>' 2026-02-22T20:32:29.798348+0000 2026-02-22T20:32:47.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:32:48.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:32:48.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:32:48.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:32:48.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:32:48.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:32:48.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:32:48.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:32:48.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:32:43.605290+0000 '>' 2026-02-22T20:32:29.798348+0000 2026-02-22T20:32:48.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:32:48.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:32:48.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:32:48.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:32:48.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-22T20:32:48.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:32:48.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:32:48.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:32:48.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:32:48.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-22T20:32:48.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:32:48.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-22T20:32:48.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:32:48.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:32:48.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:32:48.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:32:48.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:32:49.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:32:49.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:32:49.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:32:49.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:32:49.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-22T20:32:49.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:32:49.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:32:49.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-22T20:32:49.288 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:32:49.288 INFO:tasks.workunit.client.0.trial150.stdout:hinfo_key 2026-02-22T20:32:49.288 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:32:49.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-22T20:32:49.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:32:49.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:32:49.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-22T20:32:49.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:32:49.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:32:49.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:32:49.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:32:49.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:32:49.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:32:49.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-22T20:32:49.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-22T20:32:49.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:32:49.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:32:49.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:32:49.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:32:49.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:32:49.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:32:49.378 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:32:49.378 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:32:49.378 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:32:49.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:49.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:49.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:32:49.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:32:49.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:32:49.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:32:49.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:32:49.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:32:49.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:32:49.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:32:49.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:32:49.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-22T20:32:49.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-22T20:32:49.381 INFO:tasks.workunit.client.0.trial150.stderr:start osd.9 2026-02-22T20:32:49.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:32:49.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-22T20:32:49.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-22T20:32:49.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:32:49.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:32:49.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:32:49.398 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:49.384+0000 7f09c46e4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:49.399 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:49.388+0000 7f09c46e4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:49.400 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:49.388+0000 7f09c46e4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:49.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-22T20:32:49.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:32:49.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-22T20:32:49.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:32:49.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:32:49.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:49.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:32:49.539 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:32:49.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:49.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:32:49.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:49.864 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:49.852+0000 7f09c46e4a40 -1 Falling back to public interface 2026-02-22T20:32:50.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:50.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:50.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:32:50.701 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:32:50.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:50.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:32:50.812 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:50.800+0000 7f09c46e4a40 -1 osd.9 68 log_to_monitors true 2026-02-22T20:32:50.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:32:51.863 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:32:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:32:52.017 INFO:tasks.workunit.client.0.trial150.stderr:osd.9 up in weight 1 up_from 71 up_thru 0 down_at 68 last_clean_interval [46,68) [v2:127.0.0.1:6874/458073857,v1:127.0.0.1:6875/458073857] [v2:127.0.0.1:6876/458073857,v1:127.0.0.1:6877/458073857] exists,up 0871896b-7ed0-47d4-99e7-9d74df750764 2026-02-22T20:32:52.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:32:52.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:32:52.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:32:52.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:32:52.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:32:52.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:32:52.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:32:52.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:32:52.019 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:32:52.019 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:32:52.019 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:32:52.019 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:32:52.019 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:32:52.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:32:52.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:32:52.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:32:52.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:32:52.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:32:52.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:32:52.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:32:52.286 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:32:52.286 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:32:52.286 INFO:tasks.workunit.client.0.trial150.stderr:3 2026-02-22T20:32:52.286 INFO:tasks.workunit.client.0.trial150.stderr:4 2026-02-22T20:32:52.286 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:32:52.286 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:32:52.287 INFO:tasks.workunit.client.0.trial150.stderr:7 2026-02-22T20:32:52.287 INFO:tasks.workunit.client.0.trial150.stderr:8 2026-02-22T20:32:52.287 INFO:tasks.workunit.client.0.trial150.stderr:9' 2026-02-22T20:32:52.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:32:52.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:52.287 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:32:52.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-22T20:32:52.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-22T20:32:52.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-22T20:32:52.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:52.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:32:52.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672976 2026-02-22T20:32:52.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672976 2026-02-22T20:32:52.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976' 2026-02-22T20:32:52.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:52.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:32:52.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542160 2026-02-22T20:32:52.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542160 2026-02-22T20:32:52.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-60129542160' 2026-02-22T20:32:52.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:52.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:32:52.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841541 2026-02-22T20:32:52.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841541 2026-02-22T20:32:52.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-60129542160 3-283467841541' 2026-02-22T20:32:52.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:52.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:32:52.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215118 2026-02-22T20:32:52.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215118 2026-02-22T20:32:52.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-60129542160 3-283467841541 4-103079215118' 2026-02-22T20:32:52.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:52.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:32:52.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084301 2026-02-22T20:32:52.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084301 2026-02-22T20:32:52.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-60129542160 3-283467841541 4-103079215118 5-120259084301' 2026-02-22T20:32:52.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:52.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:32:52.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953485 2026-02-22T20:32:52.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953485 2026-02-22T20:32:52.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-60129542160 3-283467841541 4-103079215118 5-120259084301 6-137438953485' 2026-02-22T20:32:52.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:52.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:32:52.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789964 2026-02-22T20:32:52.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789964 2026-02-22T20:32:52.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-60129542160 3-283467841541 4-103079215118 5-120259084301 6-137438953485 7-158913789964' 2026-02-22T20:32:52.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:52.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:32:52.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626443 2026-02-22T20:32:52.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626443 2026-02-22T20:32:52.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-60129542160 3-283467841541 4-103079215118 5-120259084301 6-137438953485 7-158913789964 8-180388626443' 2026-02-22T20:32:52.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:52.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:32:53.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678018 2026-02-22T20:32:53.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678018 2026-02-22T20:32:53.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-60129542160 3-283467841541 4-103079215118 5-120259084301 6-137438953485 7-158913789964 8-180388626443 9-304942678018' 2026-02-22T20:32:53.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:53.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-22T20:32:53.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:53.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:32:53.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-22T20:32:53.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:53.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-22T20:32:53.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-22T20:32:53.010 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836497 2026-02-22T20:32:53.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:32:53.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836497 2026-02-22T20:32:53.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:53.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672976 2026-02-22T20:32:53.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:53.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:32:53.180 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672976 2026-02-22T20:32:53.180 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:53.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672976 2026-02-22T20:32:53.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2026-02-22T20:32:53.182 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672976 2026-02-22T20:32:53.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:32:53.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672976 -lt 42949672976 2026-02-22T20:32:53.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:53.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542160 2026-02-22T20:32:53.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:53.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:32:53.352 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542160 2026-02-22T20:32:53.352 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:53.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542160 2026-02-22T20:32:53.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2026-02-22T20:32:53.354 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542160 2026-02-22T20:32:53.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:32:53.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542160 -lt 60129542160 2026-02-22T20:32:53.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:53.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841541 2026-02-22T20:32:53.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:53.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:32:53.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841541 2026-02-22T20:32:53.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:53.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841541 2026-02-22T20:32:53.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841541' 2026-02-22T20:32:53.529 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 283467841541 2026-02-22T20:32:53.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:32:53.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841542 -lt 283467841541 2026-02-22T20:32:53.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:53.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215118 2026-02-22T20:32:53.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:53.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:32:53.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215118 2026-02-22T20:32:53.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:53.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215118 2026-02-22T20:32:53.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215118' 2026-02-22T20:32:53.698 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.4 seq 103079215118 2026-02-22T20:32:53.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:32:53.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215118 -lt 103079215118 2026-02-22T20:32:53.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:53.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-120259084301 2026-02-22T20:32:53.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:53.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:32:53.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-120259084301 2026-02-22T20:32:53.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:53.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084301 2026-02-22T20:32:53.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 120259084301' 2026-02-22T20:32:53.863 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.5 seq 120259084301 2026-02-22T20:32:53.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:32:54.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084301 -lt 120259084301 2026-02-22T20:32:54.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:54.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953485 2026-02-22T20:32:54.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:54.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:32:54.024 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953485 2026-02-22T20:32:54.024 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:54.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953485 2026-02-22T20:32:54.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953485' 2026-02-22T20:32:54.026 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.6 seq 137438953485 2026-02-22T20:32:54.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:32:54.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953485 -lt 137438953485 2026-02-22T20:32:54.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:54.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789964 2026-02-22T20:32:54.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:54.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:32:54.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789964 2026-02-22T20:32:54.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:54.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789964 2026-02-22T20:32:54.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789964' 2026-02-22T20:32:54.203 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.7 seq 158913789964 2026-02-22T20:32:54.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:32:54.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789963 -lt 158913789964 2026-02-22T20:32:54.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:32:55.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:32:55.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:32:55.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789964 -lt 158913789964 2026-02-22T20:32:55.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:55.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626443 2026-02-22T20:32:55.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:55.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:32:55.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626443 2026-02-22T20:32:55.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:55.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626443 2026-02-22T20:32:55.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626443' 2026-02-22T20:32:55.532 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.8 seq 180388626443 2026-02-22T20:32:55.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:32:55.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626444 -lt 180388626443 2026-02-22T20:32:55.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:55.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-304942678018 2026-02-22T20:32:55.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:55.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:32:55.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-304942678018 2026-02-22T20:32:55.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:55.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678018 2026-02-22T20:32:55.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 304942678018' 2026-02-22T20:32:55.697 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.9 seq 304942678018 2026-02-22T20:32:55.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:32:55.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678018 -lt 304942678018 2026-02-22T20:32:55.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:32:55.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:32:55.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:32:56.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:32:56.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:32:56.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:32:56.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:32:56.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:32:56.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:32:56.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:32:56.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:32:56.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:32:56.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:32:56.223 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:32:56.223 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:32:56.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:32:56.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:32:56.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:32:56.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:32:56.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:32:56.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:321: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2026-02-22T20:32:56.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-22T20:32:56.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-22T20:32:56.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=5 2026-02-22T20:32:56.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:32:56.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:32:56.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:32:56.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-22T20:32:56.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:32:56.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:32:56.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:32:56.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:32:56.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-22T20:32:56.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:32:56.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-22T20:32:56.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:32:56.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:32:56.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:32:56.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:32:56.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:32:56.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:32:56.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:32:56.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:32:56.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:32:56.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-22T20:32:56.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:32:56.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:32:56.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-22T20:32:56.919 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:32:57.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-22T20:32:57.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:32:57.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:32:57.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-22T20:32:57.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:32:57.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:32:57.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:32:57.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:32:57.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:32:57.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:32:57.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-22T20:32:57.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-22T20:32:57.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:32:57.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:32:57.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:32:57.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:32:57.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:32:57.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:32:57.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:32:57.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:32:57.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-22T20:32:57.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-22T20:32:57.022 INFO:tasks.workunit.client.0.trial150.stderr:start osd.5 2026-02-22T20:32:57.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:32:57.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-22T20:32:57.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-22T20:32:57.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:32:57.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:32:57.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:32:57.042 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:57.028+0000 7fbe82341a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:57.043 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:57.032+0000 7fbe82341a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:57.044 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:57.032+0000 7fbe82341a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:32:57.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-22T20:32:57.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:32:57.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-22T20:32:57.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:32:57.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:32:57.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:57.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:32:57.183 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:32:57.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:57.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:32:57.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:32:57.480 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:57.468+0000 7fbe82341a40 -1 Falling back to public interface 2026-02-22T20:32:58.055 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:32:58.040+0000 7fbe82341a40 -1 osd.5 72 log_to_monitors true 2026-02-22T20:32:58.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:32:58.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:32:58.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:32:58.344 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:32:58.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:32:58.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:32:58.503 INFO:tasks.workunit.client.0.trial150.stderr:osd.5 up in weight 1 up_from 76 up_thru 76 down_at 73 last_clean_interval [28,72) [v2:127.0.0.1:6842/1852716321,v1:127.0.0.1:6843/1852716321] [v2:127.0.0.1:6844/1852716321,v1:127.0.0.1:6845/1852716321] exists,up b815c525-94b6-4f0c-bea0-1363d5e04434 2026-02-22T20:32:58.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:32:58.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:32:58.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:32:58.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:32:58.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:32:58.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:32:58.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:32:58.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:32:58.504 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:32:58.505 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:32:58.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:32:58.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:32:58.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:32:58.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:32:58.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:32:58.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:32:58.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:32:58.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:32:58.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:32:58.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:32:58.778 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:32:58.778 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:32:58.778 INFO:tasks.workunit.client.0.trial150.stderr:3 2026-02-22T20:32:58.778 INFO:tasks.workunit.client.0.trial150.stderr:4 2026-02-22T20:32:58.778 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:32:58.778 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:32:58.778 INFO:tasks.workunit.client.0.trial150.stderr:7 2026-02-22T20:32:58.779 INFO:tasks.workunit.client.0.trial150.stderr:8 2026-02-22T20:32:58.779 INFO:tasks.workunit.client.0.trial150.stderr:9' 2026-02-22T20:32:58.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:32:58.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:58.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:32:58.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-22T20:32:58.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-22T20:32:58.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-22T20:32:58.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:58.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:32:58.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672979 2026-02-22T20:32:58.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672979 2026-02-22T20:32:58.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979' 2026-02-22T20:32:58.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:58.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:32:58.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542162 2026-02-22T20:32:58.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542162 2026-02-22T20:32:58.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-60129542162' 2026-02-22T20:32:58.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:58.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:32:59.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841544 2026-02-22T20:32:59.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841544 2026-02-22T20:32:59.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-60129542162 3-283467841544' 2026-02-22T20:32:59.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:59.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:32:59.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215120 2026-02-22T20:32:59.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215120 2026-02-22T20:32:59.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-60129542162 3-283467841544 4-103079215120' 2026-02-22T20:32:59.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:59.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:32:59.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=326417514498 2026-02-22T20:32:59.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 326417514498 2026-02-22T20:32:59.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-60129542162 3-283467841544 4-103079215120 5-326417514498' 2026-02-22T20:32:59.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:59.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:32:59.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953487 2026-02-22T20:32:59.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953487 2026-02-22T20:32:59.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-60129542162 3-283467841544 4-103079215120 5-326417514498 6-137438953487' 2026-02-22T20:32:59.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:59.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:32:59.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789966 2026-02-22T20:32:59.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789966 2026-02-22T20:32:59.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-60129542162 3-283467841544 4-103079215120 5-326417514498 6-137438953487 7-158913789966' 2026-02-22T20:32:59.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:59.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:32:59.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626446 2026-02-22T20:32:59.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626446 2026-02-22T20:32:59.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-60129542162 3-283467841544 4-103079215120 5-326417514498 6-137438953487 7-158913789966 8-180388626446' 2026-02-22T20:32:59.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:32:59.428 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:32:59.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=304942678020 2026-02-22T20:32:59.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 304942678020 2026-02-22T20:32:59.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672979 2-60129542162 3-283467841544 4-103079215120 5-326417514498 6-137438953487 7-158913789966 8-180388626446 9-304942678020' 2026-02-22T20:32:59.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:32:59.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-22T20:32:59.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:32:59.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:32:59.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-22T20:32:59.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:32:59.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-22T20:32:59.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-22T20:32:59.503 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836499 2026-02-22T20:32:59.503 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:32:59.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836499 2026-02-22T20:32:59.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:33:00.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:33:00.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:33:00.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836499 2026-02-22T20:33:00.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:33:01.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:33:01.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:33:01.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836499 2026-02-22T20:33:01.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:01.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672979 2026-02-22T20:33:01.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:01.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:33:01.987 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672979 2026-02-22T20:33:01.988 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:01.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672979 2026-02-22T20:33:01.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2026-02-22T20:33:01.989 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672979 2026-02-22T20:33:01.989 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:33:02.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672979 -lt 42949672979 2026-02-22T20:33:02.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:02.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542162 2026-02-22T20:33:02.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:02.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:33:02.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542162 2026-02-22T20:33:02.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:02.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542162 2026-02-22T20:33:02.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542162' 2026-02-22T20:33:02.155 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542162 2026-02-22T20:33:02.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:33:02.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542162 -lt 60129542162 2026-02-22T20:33:02.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:02.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841544 2026-02-22T20:33:02.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:02.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:33:02.327 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841544 2026-02-22T20:33:02.327 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:02.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841544 2026-02-22T20:33:02.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841544' 2026-02-22T20:33:02.329 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 283467841544 2026-02-22T20:33:02.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:33:02.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841544 -lt 283467841544 2026-02-22T20:33:02.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:02.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215120 2026-02-22T20:33:02.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:02.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:33:02.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215120 2026-02-22T20:33:02.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:02.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215120 2026-02-22T20:33:02.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215120' 2026-02-22T20:33:02.506 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.4 seq 103079215120 2026-02-22T20:33:02.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:33:02.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215121 -lt 103079215120 2026-02-22T20:33:02.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:02.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-326417514498 2026-02-22T20:33:02.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:02.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:33:02.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-326417514498 2026-02-22T20:33:02.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:02.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=326417514498 2026-02-22T20:33:02.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 326417514498' 2026-02-22T20:33:02.679 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.5 seq 326417514498 2026-02-22T20:33:02.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:33:02.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 326417514498 -lt 326417514498 2026-02-22T20:33:02.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:02.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953487 2026-02-22T20:33:02.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:02.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:33:02.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953487 2026-02-22T20:33:02.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:02.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953487 2026-02-22T20:33:02.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953487' 2026-02-22T20:33:02.844 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.6 seq 137438953487 2026-02-22T20:33:02.844 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:33:02.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953488 -lt 137438953487 2026-02-22T20:33:02.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:03.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789966 2026-02-22T20:33:03.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:03.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:33:03.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789966 2026-02-22T20:33:03.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:03.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789966 2026-02-22T20:33:03.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789966' 2026-02-22T20:33:03.004 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.7 seq 158913789966 2026-02-22T20:33:03.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:33:03.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789967 -lt 158913789966 2026-02-22T20:33:03.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:03.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626446 2026-02-22T20:33:03.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:03.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:33:03.175 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626446 2026-02-22T20:33:03.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:03.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626446 2026-02-22T20:33:03.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626446' 2026-02-22T20:33:03.177 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.8 seq 180388626446 2026-02-22T20:33:03.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:33:03.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626446 -lt 180388626446 2026-02-22T20:33:03.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:03.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-304942678020 2026-02-22T20:33:03.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:03.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:33:03.342 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-304942678020 2026-02-22T20:33:03.342 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:03.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=304942678020 2026-02-22T20:33:03.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 304942678020' 2026-02-22T20:33:03.343 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.9 seq 304942678020 2026-02-22T20:33:03.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:03.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 304942678021 -lt 304942678020 2026-02-22T20:33:03.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:33:03.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:03.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:03.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:33:03.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:33:03.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:33:03.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:33:03.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:33:03.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:33:03.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:33:03.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:33:03.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:33:03.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:33:03.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:03.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:04.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:33:04.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:33:04.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:33:04.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-22T20:33:04.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:33:04.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:33:04.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:33:04.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:33:04.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-22T20:33:04.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-22T20:33:04.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:33:04.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:33:04.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:04.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:04.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:04.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:04.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:32:43.605290+0000 2026-02-22T20:33:04.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:33:04.511 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:33:04.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:32:43.605290+0000 2026-02-22T20:33:04.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:33:04.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:32:43.605290+0000 2026-02-22T20:33:04.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:33:04.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:33:04.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:04.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:04.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:04.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:04.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:04.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:04.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:32:43.605290+0000 '>' 2026-02-22T20:32:43.605290+0000 2026-02-22T20:33:04.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:05.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:05.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:05.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:05.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:05.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:05.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:05.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:05.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:32:43.605290+0000 '>' 2026-02-22T20:32:43.605290+0000 2026-02-22T20:33:05.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:06.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:06.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:06.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:06.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:06.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:06.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:06.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:06.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:32:43.605290+0000 '>' 2026-02-22T20:32:43.605290+0000 2026-02-22T20:33:06.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:07.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:07.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:07.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:07.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:07.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:08.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:08.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:08.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:32:43.605290+0000 '>' 2026-02-22T20:32:43.605290+0000 2026-02-22T20:33:08.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:09.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:09.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:09.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:09.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:09.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:09.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:09.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:09.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:04.590985+0000 '>' 2026-02-22T20:32:43.605290+0000 2026-02-22T20:33:09.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:33:09.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:33:09.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:33:09.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:33:09.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-22T20:33:09.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:33:09.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:33:09.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:33:09.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:33:09.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-22T20:33:09.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:33:09.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-22T20:33:09.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:33:09.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:33:09.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:33:09.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:33:09.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:33:09.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:33:09.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:33:09.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:33:09.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:33:09.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-22T20:33:09.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:33:09.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:33:09.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-22T20:33:09.709 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:33:09.709 INFO:tasks.workunit.client.0.trial150.stdout:hinfo_key 2026-02-22T20:33:09.709 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:33:09.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-22T20:33:09.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:33:09.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:33:09.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-22T20:33:09.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:33:09.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:33:09.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:33:09.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:33:09.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:33:09.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:33:09.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-22T20:33:09.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-22T20:33:09.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:33:09.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:33:09.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:33:09.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:33:09.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:33:09.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:33:09.806 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:33:09.806 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:09.806 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:09.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:09.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:09.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:33:09.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:33:09.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:33:09.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:33:09.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:33:09.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:33:09.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:33:09.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:33:09.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:33:09.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-22T20:33:09.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-22T20:33:09.809 INFO:tasks.workunit.client.0.trial150.stderr:start osd.9 2026-02-22T20:33:09.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:33:09.810 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-22T20:33:09.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-22T20:33:09.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:33:09.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:33:09.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:33:09.825 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:09.812+0000 7fdc30b10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:09.826 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:09.812+0000 7fdc30b10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:09.827 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:09.816+0000 7fdc30b10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:09.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-22T20:33:09.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:33:09.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-22T20:33:09.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:33:09.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:33:09.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:09.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:33:09.966 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:33:09.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:09.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:33:10.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:33:10.212 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:10.200+0000 7fdc30b10a40 -1 Falling back to public interface 2026-02-22T20:33:10.829 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:10.816+0000 7fdc30b10a40 -1 osd.9 79 log_to_monitors true 2026-02-22T20:33:11.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:33:11.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:11.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:33:11.129 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:33:11.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:11.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:33:11.284 INFO:tasks.workunit.client.0.trial150.stderr:osd.9 up in weight 1 up_from 83 up_thru 0 down_at 80 last_clean_interval [71,79) [v2:127.0.0.1:6874/2959968702,v1:127.0.0.1:6875/2959968702] [v2:127.0.0.1:6876/2959968702,v1:127.0.0.1:6877/2959968702] exists,up 0871896b-7ed0-47d4-99e7-9d74df750764 2026-02-22T20:33:11.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:33:11.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:33:11.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:33:11.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:33:11.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:33:11.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:33:11.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:33:11.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:33:11.285 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:33:11.285 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:33:11.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:33:11.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:33:11.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:33:11.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:33:11.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:33:11.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:33:11.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:33:11.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:33:11.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:33:11.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:33:11.554 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:33:11.554 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:33:11.554 INFO:tasks.workunit.client.0.trial150.stderr:3 2026-02-22T20:33:11.554 INFO:tasks.workunit.client.0.trial150.stderr:4 2026-02-22T20:33:11.554 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:33:11.554 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:33:11.554 INFO:tasks.workunit.client.0.trial150.stderr:7 2026-02-22T20:33:11.555 INFO:tasks.workunit.client.0.trial150.stderr:8 2026-02-22T20:33:11.555 INFO:tasks.workunit.client.0.trial150.stderr:9' 2026-02-22T20:33:11.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:33:11.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:11.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:33:11.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836503 2026-02-22T20:33:11.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836503 2026-02-22T20:33:11.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503' 2026-02-22T20:33:11.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:11.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:33:11.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672982 2026-02-22T20:33:11.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672982 2026-02-22T20:33:11.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982' 2026-02-22T20:33:11.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:11.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:33:11.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542165 2026-02-22T20:33:11.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542165 2026-02-22T20:33:11.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-60129542165' 2026-02-22T20:33:11.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:11.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:33:11.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841547 2026-02-22T20:33:11.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841547 2026-02-22T20:33:11.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-60129542165 3-283467841547' 2026-02-22T20:33:11.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:11.844 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:33:11.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215124 2026-02-22T20:33:11.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215124 2026-02-22T20:33:11.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-60129542165 3-283467841547 4-103079215124' 2026-02-22T20:33:11.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:11.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:33:11.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=326417514501 2026-02-22T20:33:11.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 326417514501 2026-02-22T20:33:11.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-60129542165 3-283467841547 4-103079215124 5-326417514501' 2026-02-22T20:33:11.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:11.989 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:33:12.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953491 2026-02-22T20:33:12.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953491 2026-02-22T20:33:12.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-60129542165 3-283467841547 4-103079215124 5-326417514501 6-137438953491' 2026-02-22T20:33:12.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:12.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:33:12.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789970 2026-02-22T20:33:12.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789970 2026-02-22T20:33:12.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-60129542165 3-283467841547 4-103079215124 5-326417514501 6-137438953491 7-158913789970' 2026-02-22T20:33:12.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:12.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:33:12.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626449 2026-02-22T20:33:12.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626449 2026-02-22T20:33:12.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-60129542165 3-283467841547 4-103079215124 5-326417514501 6-137438953491 7-158913789970 8-180388626449' 2026-02-22T20:33:12.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:12.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:33:12.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=356482285570 2026-02-22T20:33:12.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 356482285570 2026-02-22T20:33:12.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672982 2-60129542165 3-283467841547 4-103079215124 5-326417514501 6-137438953491 7-158913789970 8-180388626449 9-356482285570' 2026-02-22T20:33:12.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:12.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836503 2026-02-22T20:33:12.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:12.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:33:12.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836503 2026-02-22T20:33:12.280 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:12.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836503 2026-02-22T20:33:12.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2026-02-22T20:33:12.281 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836503 2026-02-22T20:33:12.282 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:33:12.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836503 2026-02-22T20:33:12.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:33:13.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:33:13.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:33:13.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836503 2026-02-22T20:33:13.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:13.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672982 2026-02-22T20:33:13.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:13.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:33:13.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672982 2026-02-22T20:33:13.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:13.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672982 2026-02-22T20:33:13.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2026-02-22T20:33:13.610 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672982 2026-02-22T20:33:13.610 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:33:13.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672982 -lt 42949672982 2026-02-22T20:33:13.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:13.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542165 2026-02-22T20:33:13.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:13.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:33:13.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542165 2026-02-22T20:33:13.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:13.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542165 2026-02-22T20:33:13.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542165' 2026-02-22T20:33:13.780 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542165 2026-02-22T20:33:13.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:33:13.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542166 -lt 60129542165 2026-02-22T20:33:13.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:13.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841547 2026-02-22T20:33:13.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:13.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:33:13.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841547 2026-02-22T20:33:13.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:13.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841547 2026-02-22T20:33:13.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841547' 2026-02-22T20:33:13.947 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 283467841547 2026-02-22T20:33:13.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:33:14.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841548 -lt 283467841547 2026-02-22T20:33:14.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:14.106 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215124 2026-02-22T20:33:14.106 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:14.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:33:14.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215124 2026-02-22T20:33:14.109 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:14.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215124 2026-02-22T20:33:14.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215124' 2026-02-22T20:33:14.110 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.4 seq 103079215124 2026-02-22T20:33:14.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:33:14.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215124 -lt 103079215124 2026-02-22T20:33:14.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:14.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-326417514501 2026-02-22T20:33:14.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:14.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:33:14.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-326417514501 2026-02-22T20:33:14.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:14.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=326417514501 2026-02-22T20:33:14.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 326417514501' 2026-02-22T20:33:14.285 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.5 seq 326417514501 2026-02-22T20:33:14.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:33:14.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 326417514501 -lt 326417514501 2026-02-22T20:33:14.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:14.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953491 2026-02-22T20:33:14.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:14.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:33:14.455 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953491 2026-02-22T20:33:14.455 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:14.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953491 2026-02-22T20:33:14.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953491' 2026-02-22T20:33:14.457 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.6 seq 137438953491 2026-02-22T20:33:14.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:33:14.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953491 -lt 137438953491 2026-02-22T20:33:14.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:14.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789970 2026-02-22T20:33:14.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:14.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:33:14.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789970 2026-02-22T20:33:14.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:14.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789970 2026-02-22T20:33:14.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789970' 2026-02-22T20:33:14.625 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.7 seq 158913789970 2026-02-22T20:33:14.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:33:14.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789970 -lt 158913789970 2026-02-22T20:33:14.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:14.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626449 2026-02-22T20:33:14.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:14.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:33:14.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626449 2026-02-22T20:33:14.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:14.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626449 2026-02-22T20:33:14.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626449' 2026-02-22T20:33:14.799 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.8 seq 180388626449 2026-02-22T20:33:14.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:33:14.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626450 -lt 180388626449 2026-02-22T20:33:14.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:14.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-356482285570 2026-02-22T20:33:14.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:14.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:33:14.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-356482285570 2026-02-22T20:33:14.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:14.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=356482285570 2026-02-22T20:33:14.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 356482285570' 2026-02-22T20:33:14.962 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.9 seq 356482285570 2026-02-22T20:33:14.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:15.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 356482285570 -lt 356482285570 2026-02-22T20:33:15.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:33:15.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:15.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:15.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:33:15.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:33:15.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:33:15.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:33:15.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:33:15.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:33:15.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:33:15.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:33:15.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:33:15.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:33:15.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:15.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:15.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:33:15.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:33:15.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:33:15.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:33:15.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:33:15.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2026-02-22T20:33:15.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-22T20:33:15.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-22T20:33:15.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=5 2026-02-22T20:33:15.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=9 2026-02-22T20:33:15.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-22T20:33:15.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:33:15.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:33:15.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:33:15.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 189457"' 2026-02-22T20:33:15.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 189457' 2026-02-22T20:33:15.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2026-02-22T20:33:15.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:33:15.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:33:15.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 189459"' 2026-02-22T20:33:15.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:33:15.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 189459' 2026-02-22T20:33:15.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-22T20:33:15.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 189457 189459' 2026-02-22T20:33:15.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:33:15.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:33:15.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 189457 2026-02-22T20:33:15.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/189460: /' 2026-02-22T20:33:15.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2026-02-22T20:33:15.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/189463: /' 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: remove 2#2:eb822e21:::SOMETHING:head# 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-22T20:33:16.311 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:33:16.312 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:33:16.312 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-22T20:33:16.312 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:33:16.312 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:33:16.312 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:33:16.312 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:33:16.314 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:33:16.314 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:33:16.314 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-22T20:33:16.314 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-22T20:33:16.314 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:33:16.314 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:33:16.314 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:33:16.314 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:33:16.314 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:33:16.314 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: start osd.9 2026-02-22T20:33:16.315 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:33:16.359 INFO:tasks.workunit.client.0.trial150.stderr:1189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:33:16.360 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:33:16.361 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:33:16.362 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:33:16.363 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:33:16.363 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:33:16.363 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-22T20:33:16.363 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-22T20:33:16.363 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:33:16.363 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:33:16.363 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:33:16.363 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:33:16.363 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:33:16.363 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:33:16.363 INFO:tasks.workunit.client.0.trial150.stderr:189460: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-22T20:33:16.364 INFO:tasks.workunit.client.0.trial150.stderr:189460: start osd.5 2026-02-22T20:33:16.365 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:33:17.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-22T20:33:17.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-22T20:33:17.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:33:17.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:33:17.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:33:17.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: 2026-02-22T20:33:16.320+0000 7ff4d7177a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:17.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: 2026-02-22T20:33:16.320+0000 7ff4d7177a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: 2026-02-22T20:33:16.324+0000 7ff4d7177a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: 0 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: 2026-02-22T20:33:16.808+0000 7ff4d7177a40 -1 Falling back to public interface 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: 2026-02-22T20:33:17.508+0000 7ff4d7177a40 -1 osd.9 84 log_to_monitors true 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: 1 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: osd.9 up in weight 1 up_from 88 up_thru 0 down_at 85 last_clean_interval [83,84) [v2:127.0.0.1:6842/1516526240,v1:127.0.0.1:6843/1516526240] [v2:127.0.0.1:6844/1516526240,v1:127.0.0.1:6845/1516526240] exists,up 0871896b-7ed0-47d4-99e7-9d74df750764 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:33:17.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:33:17.854 INFO:tasks.workunit.client.0.trial150.stderr:189463: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: ge89460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-22T20:33:17.854 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-22T20:33:17.854 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:33:17.854 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:33:17.854 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:33:17.854 INFO:tasks.workunit.client.0.trial150.stderr:189460: 2026-02-22T20:33:16.364+0000 7f0550fa4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:17.854 INFO:tasks.workunit.client.0.trial150.stderr:189460: 2026-02-22T20:33:16.368+0000 7f0550fa4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:17.854 INFO:tasks.workunit.client.0.trial150.stderr:189460: 2026-02-22T20:33:16.368+0000 7f0550fa4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:17.854 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-22T20:33:17.854 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: 0 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: 2026-02-22T20:33:16.872+0000 7f0550fa4a40 -1 Falling back to public interface 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: 2026-02-22T20:33:17.576+0000 7f0550fa4a40 -1 osd.5 85 log_to_monitors true 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: 1 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:17.855 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:33:17.856 INFO:tasks.workunit.client.0.trial150.stderr:189460: osd.5 up in weight 1 up_from 89 up_thru 76 down_at 85 last_clean_interval [76,85) [v2:127.0.0.1:6874/221652766,v1:127.0.0.1:6875/221652766] [v2:127.0.0.1:6876/221652766,v1:127.0.0.1:6877/221652766] exists,up b815c525-94b6-4f0c-bea0-1363d5e04434 2026-02-22T20:33:17.856 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:33:17.856 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:33:17.856 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:33:17.856 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:33:17.856 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:33:17.856 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:33:17.856 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:33:17.856 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:33:17.856 INFO:tasks.workunit.client.0.trial150.stderr:189460: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:33:18.362 INFO:tasks.workunit.client.0.trial150.stderr:189460: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_tt_timeout_delays: echo true 2026-02-22T20:33:18.363 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:33:18.363 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:33:18.363 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:33:18.363 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:33:18.363 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:33:18.363 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:33:18.363 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:33:18.364 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:33:18.364 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:33:18.364 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:33:18.364 INFO:tasks.workunit.client.0.trial150.stderr:189463: 1 2026-02-22T20:33:18.364 INFO:tasks.workunit.client.0.trial150.stderr:189463: 2 2026-02-22T20:33:18.364 INFO:tasks.workunit.client.0.trial150.stderr:189463: 3 2026-02-22T20:33:18.364 INFO:tasks.workunit.client.0.trial150.stderr:189463: 4 2026-02-22T20:33:18.364 INFO:tasks.workunit.client.0.trial150.stderr:189463: 5 2026-02-22T20:33:18.364 INFO:tasks.workunit.client.0.trial150.stderr:189463: 6 2026-02-22T20:33:18.364 INFO:tasks.workunit.client.0.trial150.stderr:189463: 7 2026-02-22T20:33:18.364 INFO:tasks.workunit.client.0.trial150.stderr:189463: 8 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: 9' 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836505 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836505 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505' 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672984 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672984 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672984' 2026-02-22T20:33:18.365 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.366 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:33:18.366 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542168 2026-02-22T20:33:18.366 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542168 2026-02-22T20:33:18.366 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-60129542168' 2026-02-22T20:33:18.366 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.366 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:33:18.366 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841550 2026-02-22T20:33:18.366 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841550 2026-02-22T20:33:18.366 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-60129542168 3-283467841550' 2026-02-22T20:33:18.366 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.436 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: imeout_delays: echo true 2026-02-22T20:33:18.436 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:33:18.436 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:33:18.436 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:33:18.436 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:33:18.436 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: 1 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: 2 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: 3 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: 4 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: 5 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: 6 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: 7 2026-02-22T20:33:18.437 INFO:tasks.workunit.client.0.trial150.stderr:189460: 8 2026-02-22T20:33:18.438 INFO:tasks.workunit.client.0.trial150.stderr:189460: 9' 2026-02-22T20:33:18.438 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:33:18.438 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.438 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:33:18.438 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836506 2026-02-22T20:33:18.438 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836506 2026-02-22T20:33:18.438 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506' 2026-02-22T20:33:18.438 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.438 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:33:18.438 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672985 2026-02-22T20:33:18.438 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672985 2026-02-22T20:33:18.439 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672985' 2026-02-22T20:33:18.439 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.439 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:33:18.439 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542169 2026-02-22T20:33:18.439 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542169 2026-02-22T20:33:18.439 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-60129542169' 2026-02-22T20:33:18.439 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.439 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:33:18.439 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841551 2026-02-22T20:33:18.439 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841551 2026-02-22T20:33:18.439 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-60129542169 3-283467841551' 2026-02-22T20:33:18.439 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.802 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: fluflush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:33:18.802 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215126 2026-02-22T20:33:18.802 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215126 2026-02-22T20:33:18.802 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-60129542168 3-283467841550 4-103079215126' 2026-02-22T20:33:18.803 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.803 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:33:18.803 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089346 2026-02-22T20:33:18.803 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089346 2026-02-22T20:33:18.803 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-60129542168 3-283467841550 4-103079215126 5-382252089346' 2026-02-22T20:33:18.803 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.803 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:33:18.803 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953493 2026-02-22T20:33:18.803 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953493 2026-02-22T20:33:18.803 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-60129542168 3-283467841550 4-103079215126 5-382252089346 6-137438953493' 2026-02-22T20:33:18.803 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.803 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:33:18.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789972 2026-02-22T20:33:18.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789972 2026-02-22T20:33:18.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-60129542168 3-283467841550 4-103079215126 5-382252089346 6-137438953493 7-158913789972' 2026-02-22T20:33:18.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:33:18.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626451 2026-02-22T20:33:18.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626451 2026-02-22T20:33:18.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-60129542168 3-283467841550 4-103079215126 5-382252089346 6-137438953493 7-158913789972 8-180388626451' 2026-02-22T20:33:18.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:33:18.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122050 2026-02-22T20:33:18.804 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122050 2026-02-22T20:33:18.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-60129542168 3-283467841550 4-103079215126 5-382252089346 6-137438953493 7-158913789972 8-180388626451 9-377957122050' 2026-02-22T20:33:18.805 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:18.875 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215127 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215127 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-60129542169 3-283467841551 4-103079215127' 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089347 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089347 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-60129542169 3-283467841551 4-103079215127 5-382252089347' 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953494 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953494 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-60129542169 3-283467841551 4-103079215127 5-382252089347 6-137438953494' 2026-02-22T20:33:18.876 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789973 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789973 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-60129542169 3-283467841551 4-103079215127 5-382252089347 6-137438953494 7-158913789973' 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626452 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626452 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-60129542169 3-283467841551 4-103079215127 5-382252089347 6-137438953494 7-158913789973 8-180388626452' 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122051 2026-02-22T20:33:18.877 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122051 2026-02-22T20:33:18.878 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836506 1-42949672985 2-60129542169 3-283467841551 4-103079215127 5-382252089347 6-137438953494 7-158913789973 8-180388626452 9-377957122051' 2026-02-22T20:33:18.878 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.294 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836505 2026-02-22T20:33:19.294 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.294 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:33:19.294 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836505 2026-02-22T20:33:19.294 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.294 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836505 2026-02-22T20:33:19.294 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2026-02-22T20:33:19.294 INFO:tasks.workunit.client.0.trial150.stderr:189463: waiting osd.0 seq 21474836505 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836506 -lt 21474836505 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672984 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672984 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672984 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: waiting osd.1 seq 42949672984 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672986 -lt 42949672984 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542168 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.295 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:33:19.296 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542168 2026-02-22T20:33:19.296 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.296 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542168 2026-02-22T20:33:19.296 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542168' 2026-02-22T20:33:19.296 INFO:tasks.workunit.client.0.trial150.stderr:189463: waiting osd.2 seq 60129542168 2026-02-22T20:33:19.296 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:33:19.296 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542169 -lt 60129542168 2026-02-22T20:33:19.296 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.296 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841550 2026-02-22T20:33:19.296 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.296 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:33:19.372 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836506 2026-02-22T20:33:19.372 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.372 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:33:19.372 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836506 2026-02-22T20:33:19.372 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.372 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836506 2026-02-22T20:33:19.373 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2026-02-22T20:33:19.373 INFO:tasks.workunit.client.0.trial150.stderr:189460: waiting osd.0 seq 21474836506 2026-02-22T20:33:19.373 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:33:19.373 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836506 -lt 21474836506 2026-02-22T20:33:19.373 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.373 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672985 2026-02-22T20:33:19.373 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.373 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:33:19.373 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672985 2026-02-22T20:33:19.373 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.373 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672985 2026-02-22T20:33:19.373 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2026-02-22T20:33:19.374 INFO:tasks.workunit.client.0.trial150.stderr:189460: waiting osd.1 seq 42949672985 2026-02-22T20:33:19.374 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:33:19.374 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672986 -lt 42949672985 2026-02-22T20:33:19.374 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.374 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542169 2026-02-22T20:33:19.374 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.374 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:33:19.374 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542169 2026-02-22T20:33:19.374 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.374 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542169 2026-02-22T20:33:19.374 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542169' 2026-02-22T20:33:19.374 INFO:tasks.workunit.client.0.trial150.stderr:189460: waiting osd.2 seq 60129542169 2026-02-22T20:33:19.375 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:33:19.375 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542169 -lt 60129542169 2026-02-22T20:33:19.375 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.375 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841551 2026-02-22T20:33:19.375 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.375 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:33:19.784 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841551550 2026-02-22T20:33:19.784 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.784 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841550 2026-02-22T20:33:19.784 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841550' 2026-02-22T20:33:19.784 INFO:tasks.workunit.client.0.trial150.stderr:189463: waiting osd.3 seq 283467841550 2026-02-22T20:33:19.784 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:33:19.785 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841551 -lt 283467841550 2026-02-22T20:33:19.785 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.785 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215126 2026-02-22T20:33:19.785 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.785 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:33:19.785 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215126 2026-02-22T20:33:19.785 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.785 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215126 2026-02-22T20:33:19.785 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215126' 2026-02-22T20:33:19.785 INFO:tasks.workunit.client.0.trial150.stderr:189463: waiting osd.4 seq 103079215126 2026-02-22T20:33:19.785 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:33:19.785 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215127 -lt 103079215126 2026-02-22T20:33:19.786 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.786 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-382252089346 2026-02-22T20:33:19.786 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.786 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:33:19.786 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-382252089346 2026-02-22T20:33:19.786 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.786 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089346 2026-02-22T20:33:19.786 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 382252089346' 2026-02-22T20:33:19.786 INFO:tasks.workunit.client.0.trial150.stderr:189463: waiting osd.5 seq 382252089346 2026-02-22T20:33:19.786 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:33:19.786 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089347 -lt 382252089346 2026-02-22T20:33:19.786 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.787 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953493 2026-02-22T20:33:19.787 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.787 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:33:19.787 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953493 2026-02-22T20:33:19.787 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.787 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953493 2026-02-22T20:33:19.787 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953493' 2026-02-22T20:33:19.787 INFO:tasks.workunit.client.0.trial150.stderr:189463: waiting osd.6 seq 137438953493 2026-02-22T20:33:19.869 INFO:tasks.workunit.client.0.trial150.stderr:18946 2026-02-22T20:33:19.870 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.870 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841551 2026-02-22T20:33:19.870 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841551' 2026-02-22T20:33:19.870 INFO:tasks.workunit.client.0.trial150.stderr:189460: waiting osd.3 seq 283467841551 2026-02-22T20:33:19.870 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:33:19.870 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841551 -lt 283467841551 2026-02-22T20:33:19.870 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.870 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215127 2026-02-22T20:33:19.871 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.871 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:33:19.871 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215127 2026-02-22T20:33:19.871 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.871 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215127 2026-02-22T20:33:19.871 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215127' 2026-02-22T20:33:19.871 INFO:tasks.workunit.client.0.trial150.stderr:189460: waiting osd.4 seq 103079215127 2026-02-22T20:33:19.871 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:33:19.871 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215127 -lt 103079215127 2026-02-22T20:33:19.871 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-382252089347 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-382252089347 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089347 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 382252089347' 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: waiting osd.5 seq 382252089347 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089347 -lt 382252089347 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953494 2026-02-22T20:33:19.872 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:19.873 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:33:19.873 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953494 2026-02-22T20:33:19.873 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:19.873 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953494 2026-02-22T20:33:19.873 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953494' 2026-02-22T20:33:19.873 INFO:tasks.workunit.client.0.trial150.stderr:189460: waiting osd.6 seq 137438953494 2026-02-22T20:33:20.428 INFO:tasks.workunit.client.0.trial150.stderr:189460: 3: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:33:20.428 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953494 -lt 137438953493 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789972 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789972 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789972 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789972' 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: waiting osd.7 seq 158913789972 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789973 -lt 158913789972 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626451 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626451 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626451 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626451' 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: waiting osd.8 seq 180388626451 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626452 -lt 180388626451 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-377957122050 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-377957122050 2026-02-22T20:33:20.429 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:20.430 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122050 2026-02-22T20:33:20.430 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 377957122050' 2026-02-22T20:33:20.430 INFO:tasks.workunit.client.0.trial150.stderr:189463: waiting osd.9 seq 377957122050 2026-02-22T20:33:20.430 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:20.430 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122050 -lt 377957122050 2026-02-22T20:33:20.430 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:33:20.985 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:20.986 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:20.986 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:33:20.986 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:33:20.986 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:33:20.986 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:33:20.986 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:33:20.986 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:33:20.986 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:33:20.986 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:33:20.987 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:33:20.987 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:33:20.987 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:20.987 INFO:tasks.workunit.client.0.trial150.stderr:189463: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:20.987 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:33:20.987 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:33:20.987 INFO:tasks.workunit.client.0.trial150.stderr:189463: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:33:20.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:33:21.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:33:21.539 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953494 -lt 137438953494 2026-02-22T20:33:21.539 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:21.539 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789973 2026-02-22T20:33:21.539 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:21.539 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:33:21.539 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789973 2026-02-22T20:33:21.539 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789973 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789973' 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: waiting osd.7 seq 158913789973 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789973 -lt 158913789973 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626452 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626452 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626452 2026-02-22T20:33:21.540 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626452' 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: waiting osd.8 seq 180388626452 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626452 -lt 180388626452 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-377957122051 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-377957122051 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122051 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 377957122051' 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: waiting osd.9 seq 377957122051 2026-02-22T20:33:21.541 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:21.542 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122050 -lt 377957122051 2026-02-22T20:33:21.542 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:33:22.249 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:33:22.249 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:22.249 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122051 -lt 377957122051 2026-02-22T20:33:22.250 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:33:22.250 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:22.250 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:22.250 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:33:22.250 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:33:22.250 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:33:22.250 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:33:22.250 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:33:22.250 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:33:22.250 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:33:22.250 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr:189460: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr:189460: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 189459 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:33:22.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:33:22.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-22T20:33:22.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:33:22.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-22T20:33:22.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:33:22.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:33:22.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:33:22.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:33:22.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-22T20:33:22.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-22T20:33:22.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:33:22.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:33:22.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:22.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:22.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:22.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:22.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:33:04.590985+0000 2026-02-22T20:33:22.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:33:22.715 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:33:22.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:33:04.590985+0000 2026-02-22T20:33:22.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:33:22.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:33:04.590985+0000 2026-02-22T20:33:22.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:33:22.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:33:22.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:22.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:22.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:22.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:22.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:22.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:22.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:04.590985+0000 '>' 2026-02-22T20:33:04.590985+0000 2026-02-22T20:33:22.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:23.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:23.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:23.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:23.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:23.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:23.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:23.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:24.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:04.590985+0000 '>' 2026-02-22T20:33:04.590985+0000 2026-02-22T20:33:24.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:25.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:25.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:25.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:25.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:25.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:25.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:25.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:25.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:04.590985+0000 '>' 2026-02-22T20:33:04.590985+0000 2026-02-22T20:33:25.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:26.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:26.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:26.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:26.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:26.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:26.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:26.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:26.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:04.590985+0000 '>' 2026-02-22T20:33:04.590985+0000 2026-02-22T20:33:26.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:27.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:27.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:27.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:27.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:27.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:27.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:27.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:27.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:04.590985+0000 '>' 2026-02-22T20:33:04.590985+0000 2026-02-22T20:33:27.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:28.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:28.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:28.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:28.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:28.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:28.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:28.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:28.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:04.590985+0000 '>' 2026-02-22T20:33:04.590985+0000 2026-02-22T20:33:28.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:29.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:29.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:29.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:29.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:29.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:29.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:29.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:29.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:23.690461+0000 '>' 2026-02-22T20:33:04.590985+0000 2026-02-22T20:33:29.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:33:29.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-22T20:33:29.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:33:29.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:33:29.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:33:29.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 192896"' 2026-02-22T20:33:29.830 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 192896' 2026-02-22T20:33:29.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:33:29.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:33:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:33:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 192897"' 2026-02-22T20:33:29.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 192897' 2026-02-22T20:33:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-22T20:33:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 192896 192897' 2026-02-22T20:33:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:33:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:33:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 192896 2026-02-22T20:33:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:33:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/192899: /' 2026-02-22T20:33:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:33:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/192901: /' 2026-02-22T20:33:30.347 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:33:30.347 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:33:30.347 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-22T20:33:30.348 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:33:30.348 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:33:30.348 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:33:30.348 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:33:30.348 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-22T20:33:30.348 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:33:30.348 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-22T20:33:30.348 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:33:30.348 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:33:30.348 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:33:30.348 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:33:30.348 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:33:30.349 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:33:30.349 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:33:30.349 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:33:30.349 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:33:30.349 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-22T20:33:30.349 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:33:30.349 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:33:30.349 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2026-02-22T20:33:30.349 INFO:tasks.workunit.client.0.trial150.stderr:192899: _ 2026-02-22T20:33:30.349 INFO:tasks.workunit.client.0.trial150.stderr:192899: hinfo_key 2026-02-22T20:33:30.349 INFO:tasks.workunit.client.0.trial150.stderr:192899: snapset 2026-02-22T20:33:30.349 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-22T20:33:30.350 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:33:30.350 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:33:30.350 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-22T20:33:30.350 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:33:30.350 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:33:30.350 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:33:30.350 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:33:30.351 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:33:30.351 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:33:30.351 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-22T20:33:30.351 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-22T20:33:30.351 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:33:30.351 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:33:30.351 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:33:30.351 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:33:30.351 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:33:30.351 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:33:30.352 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:33:30.353 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:33:30.353 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-22T20:33:30.353 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-22T20:33:30.353 INFO:tasks.workunit.client.0.trial150.stderr:192899: start osd.5 2026-02-22T20:33:30.379 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:33:30.380 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:33:30.380 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-22T20:33:30.380 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:33:30.380 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:33:30.380 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:33:30.380 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:33:30.380 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-22T20:33:30.380 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:33:30.380 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-22T20:33:30.380 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:33:30.380 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: _ 2026-02-22T20:33:30.381 INFO:tasks.workunit.client.0.trial150.stderr:192901: hinfo_key 2026-02-22T20:33:30.382 INFO:tasks.workunit.client.0.trial150.stderr:192901: snapset 2026-02-22T20:33:30.382 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-22T20:33:30.382 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:33:30.382 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:33:30.382 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-22T20:33:30.382 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:33:30.382 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:33:30.382 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:33:30.382 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:33:30.383 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:33:30.383 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:33:30.383 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-22T20:33:30.383 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-22T20:33:30.383 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:33:30.383 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:33:30.383 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:33:30.383 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:33:30.383 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:33:30.384 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:33:30.384 INFO:tasks.workunit.client.0.trial150.stderr:192901: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:33:30.384 INFO:tasks.workunit.client.0.trial150.stderr:192901: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:30.384 INFO:tasks.workunit.client.0.trial150.stderr:192901: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:30.384 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:30.384 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:30.384 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:33:30.384 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:33:30.384 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:33:30.384 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:33:30.385 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:33:30.385 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:33:30.385 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:33:30.385 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:33:30.385 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:33:30.385 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-22T20:33:30.385 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-22T20:33:30.385 INFO:tasks.workunit.client.0.trial150.stderr:192901: start osd.9 2026-02-22T20:33:31.831 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:33:31.831 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-22T20:33:31.831 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-22T20:33:31.831 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:33:31.831 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:33:31.832 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:33:31.832 INFO:tasks.workunit.client.0.trial150.stderr:192899: 2026-02-22T20:33:30.352+0000 7f56c3e59a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:31.832 INFO:tasks.workunit.client.0.trial150.stderr:192899: 2026-02-22T20:33:30.356+0000 7f56c3e59a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:31.832 INFO:tasks.workunit.client.0.trial150.stderr:192899: 2026-02-22T20:33:30.356+0000 7f56c3e59a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:31.832 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-22T20:33:31.832 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:33:31.832 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-22T20:33:31.832 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:33:31.832 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:33:31.832 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:31.832 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:33:31.832 INFO:tasks.workunit.client.0.trial150.stderr:192899: 0 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: 2026-02-22T20:33:30.828+0000 7f56c3e59a40 -1 Falling back to public interface 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: 2026-02-22T20:33:31.500+0000 7f56c3e59a40 -1 osd.5 90 log_to_monitors true 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: 1 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: osd.5 up in weight 1 up_from 95 up_thru 95 down_at 91 last_clean_interval [89,90) [v2:127.0.0.1:6874/1019762828,v1:127.0.0.1:6875/1019762828] [v2:127.0.0.1:6876/1019762828,v1:127.0.0.1:6877/1019762828] exists,up b815c525-94b6-4f0c-bea0-1363d5e04434 2026-02-22T20:33:31.833 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:33:31.834 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:33:31.834 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:33:31.834 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:33:31.834 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:33:31.834 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:33:31.834 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:33:31.834 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:33:31.834 INFO:tasks.workunit.client.0.trial150.stderr:192899: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:33:31.866 INFO:tasks.workunit.client.0.trial150.stderr:192899: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.very_ops 2026-02-22T20:33:31.867 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-22T20:33:31.867 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-22T20:33:31.867 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:33:31.867 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:33:31.867 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:33:31.867 INFO:tasks.workunit.client.0.trial150.stderr:192901: 2026-02-22T20:33:30.388+0000 7fbedfbcba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:31.867 INFO:tasks.workunit.client.0.trial150.stderr:192901: 2026-02-22T20:33:30.388+0000 7fbedfbcba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:31.867 INFO:tasks.workunit.client.0.trial150.stderr:192901: 2026-02-22T20:33:30.388+0000 7fbedfbcba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:31.867 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-22T20:33:31.867 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:33:31.867 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: 0 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: 2026-02-22T20:33:30.784+0000 7fbedfbcba40 -1 Falling back to public interface 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: 2026-02-22T20:33:31.424+0000 7fbedfbcba40 -1 osd.9 90 log_to_monitors true 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:31.868 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:33:31.869 INFO:tasks.workunit.client.0.trial150.stderr:192901: 1 2026-02-22T20:33:31.869 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:31.869 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:33:31.869 INFO:tasks.workunit.client.0.trial150.stderr:192901: osd.9 up in weight 1 up_from 94 up_thru 0 down_at 91 last_clean_interval [88,90) [v2:127.0.0.1:6842/614235478,v1:127.0.0.1:6843/614235478] [v2:127.0.0.1:6844/614235478,v1:127.0.0.1:6845/614235478] exists,up 0871896b-7ed0-47d4-99e7-9d74df750764 2026-02-22T20:33:31.869 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:33:31.869 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:33:31.869 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:33:31.869 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:33:31.869 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:33:31.869 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:33:31.869 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:33:31.869 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:33:31.870 INFO:tasks.workunit.client.0.trial150.stderr:192901: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:33:32.384 INFO:tasks.workunit.client.0.trial150.stderr:192901: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:15sh:1593: get_timeout_delays: echo true 2026-02-22T20:33:32.384 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:33:32.384 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:33:32.385 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:33:32.385 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:33:32.385 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:33:32.385 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:33:32.385 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:33:32.385 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:33:32.385 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:33:32.385 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:33:32.385 INFO:tasks.workunit.client.0.trial150.stderr:192899: 1 2026-02-22T20:33:32.385 INFO:tasks.workunit.client.0.trial150.stderr:192899: 2 2026-02-22T20:33:32.385 INFO:tasks.workunit.client.0.trial150.stderr:192899: 3 2026-02-22T20:33:32.385 INFO:tasks.workunit.client.0.trial150.stderr:192899: 4 2026-02-22T20:33:32.386 INFO:tasks.workunit.client.0.trial150.stderr:192899: 5 2026-02-22T20:33:32.386 INFO:tasks.workunit.client.0.trial150.stderr:192899: 6 2026-02-22T20:33:32.386 INFO:tasks.workunit.client.0.trial150.stderr:192899: 7 2026-02-22T20:33:32.386 INFO:tasks.workunit.client.0.trial150.stderr:192899: 8 2026-02-22T20:33:32.386 INFO:tasks.workunit.client.0.trial150.stderr:192899: 9' 2026-02-22T20:33:32.386 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:33:32.386 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.386 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:33:32.386 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836510 2026-02-22T20:33:32.386 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836510 2026-02-22T20:33:32.386 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510' 2026-02-22T20:33:32.386 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.387 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:33:32.387 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672989 2026-02-22T20:33:32.387 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672989 2026-02-22T20:33:32.387 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989' 2026-02-22T20:33:32.387 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.387 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:33:32.387 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542173 2026-02-22T20:33:32.387 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542173 2026-02-22T20:33:32.387 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-60129542173' 2026-02-22T20:33:32.387 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.387 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:33:32.387 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841554 2026-02-22T20:33:32.388 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841554 2026-02-22T20:33:32.388 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-60129542173 3-283467841554' 2026-02-22T20:33:32.388 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.415 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper93: get_timeout_delays: echo true 2026-02-22T20:33:32.416 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:33:32.416 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:33:32.416 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:33:32.416 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:33:32.416 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:33:32.416 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:33:32.416 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:33:32.416 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:33:32.416 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:33:32.416 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:33:32.416 INFO:tasks.workunit.client.0.trial150.stderr:192901: 1 2026-02-22T20:33:32.417 INFO:tasks.workunit.client.0.trial150.stderr:192901: 2 2026-02-22T20:33:32.417 INFO:tasks.workunit.client.0.trial150.stderr:192901: 3 2026-02-22T20:33:32.417 INFO:tasks.workunit.client.0.trial150.stderr:192901: 4 2026-02-22T20:33:32.417 INFO:tasks.workunit.client.0.trial150.stderr:192901: 5 2026-02-22T20:33:32.417 INFO:tasks.workunit.client.0.trial150.stderr:192901: 6 2026-02-22T20:33:32.417 INFO:tasks.workunit.client.0.trial150.stderr:192901: 7 2026-02-22T20:33:32.417 INFO:tasks.workunit.client.0.trial150.stderr:192901: 8 2026-02-22T20:33:32.417 INFO:tasks.workunit.client.0.trial150.stderr:192901: 9' 2026-02-22T20:33:32.417 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:33:32.417 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.417 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:33:32.417 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836511 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836511 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511' 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672990 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672990 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990' 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542174 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542174 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-60129542174' 2026-02-22T20:33:32.418 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.419 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:33:32.419 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841555 2026-02-22T20:33:32.419 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841555 2026-02-22T20:33:32.419 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-60129542174 3-283467841555' 2026-02-22T20:33:32.419 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.822 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:s.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:33:32.822 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215131 2026-02-22T20:33:32.822 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215131 2026-02-22T20:33:32.822 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-60129542173 3-283467841554 4-103079215131' 2026-02-22T20:33:32.822 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.822 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:33:32.822 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893122 2026-02-22T20:33:32.822 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893122 2026-02-22T20:33:32.822 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-60129542173 3-283467841554 4-103079215131 5-408021893122' 2026-02-22T20:33:32.823 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.823 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:33:32.823 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953498 2026-02-22T20:33:32.823 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953498 2026-02-22T20:33:32.823 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-60129542173 3-283467841554 4-103079215131 5-408021893122 6-137438953498' 2026-02-22T20:33:32.823 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.823 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:33:32.823 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789977 2026-02-22T20:33:32.823 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789977 2026-02-22T20:33:32.823 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-60129542173 3-283467841554 4-103079215131 5-408021893122 6-137438953498 7-158913789977' 2026-02-22T20:33:32.823 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.823 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:33:32.824 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626456 2026-02-22T20:33:32.824 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626456 2026-02-22T20:33:32.824 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-60129542173 3-283467841554 4-103079215131 5-408021893122 6-137438953498 7-158913789977 8-180388626456' 2026-02-22T20:33:32.824 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.824 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:33:32.824 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925826 2026-02-22T20:33:32.824 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925826 2026-02-22T20:33:32.824 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-60129542173 3-283467841554 4-103079215131 5-408021893122 6-137438953498 7-158913789977 8-180388626456 9-403726925826' 2026-02-22T20:33:32.824 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:32.842 INFO:tasks.workunit.client.0.trial150.stderr:192899:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:33:32.842 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215132 2026-02-22T20:33:32.842 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215132 2026-02-22T20:33:32.842 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-60129542174 3-283467841555 4-103079215132' 2026-02-22T20:33:32.842 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.842 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:33:32.842 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893123 2026-02-22T20:33:32.843 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893123 2026-02-22T20:33:32.843 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-60129542174 3-283467841555 4-103079215132 5-408021893123' 2026-02-22T20:33:32.843 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.843 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:33:32.843 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953499 2026-02-22T20:33:32.843 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953499 2026-02-22T20:33:32.843 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-60129542174 3-283467841555 4-103079215132 5-408021893123 6-137438953499' 2026-02-22T20:33:32.843 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.843 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:33:32.843 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789978 2026-02-22T20:33:32.843 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789978 2026-02-22T20:33:32.843 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-60129542174 3-283467841555 4-103079215132 5-408021893123 6-137438953499 7-158913789978' 2026-02-22T20:33:32.844 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.844 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:33:32.844 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626457 2026-02-22T20:33:32.844 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626457 2026-02-22T20:33:32.844 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-60129542174 3-283467841555 4-103079215132 5-408021893123 6-137438953499 7-158913789978 8-180388626457' 2026-02-22T20:33:32.844 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:32.844 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:33:32.844 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925827 2026-02-22T20:33:32.844 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925827 2026-02-22T20:33:32.844 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836511 1-42949672990 2-60129542174 3-283467841555 4-103079215132 5-408021893123 6-137438953499 7-158913789978 8-180388626457 9-403726925827' 2026-02-22T20:33:32.844 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192901: //ho //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836510 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836510 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836510 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: waiting osd.0 seq 21474836510 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836511 -lt 21474836510 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672989 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672989 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672989 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: waiting osd.1 seq 42949672989 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672990 -lt 42949672989 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542173 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542173 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542173 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542173' 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: waiting osd.2 seq 60129542173 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542174 -lt 60129542173 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841554 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.329 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:33:33.338 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo me/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836511 2026-02-22T20:33:33.338 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.338 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:33:33.338 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836511 2026-02-22T20:33:33.338 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.338 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836511 2026-02-22T20:33:33.338 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2026-02-22T20:33:33.338 INFO:tasks.workunit.client.0.trial150.stderr:192901: waiting osd.0 seq 21474836511 2026-02-22T20:33:33.338 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:33:33.338 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836511 -lt 21474836511 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672990 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672990 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672990 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: waiting osd.1 seq 42949672990 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672990 -lt 42949672990 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542174 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542174 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542174 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542174' 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: waiting osd.2 seq 60129542174 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542174 -lt 60129542174 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-283467841555 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.339 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-283467841555 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841555 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841555' 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: waiting osd.3 seq 283467841555 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841556 -lt 283467841555 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215132 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215132 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215132 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215132' 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: waiting osd.4 seq 103079215132 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215132 -lt 103079215132 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-408021893123 2026-02-22T20:33:33.829 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-408021893123 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893123 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 408021893123' 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: waiting osd.5 seq 408021893123 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893123 -lt 408021893123 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953499 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953499 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953499 2026-02-22T20:33:33.830 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953499' 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192901: waiting osd.6 seq 1374389534993-283467841554 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841554 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 283467841554' 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: waiting osd.3 seq 283467841554 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841556 -lt 283467841554 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215131 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215131 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215131 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215131' 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: waiting osd.4 seq 103079215131 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215132 -lt 103079215131 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-408021893122 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-408021893122 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893122 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 408021893122' 2026-02-22T20:33:33.836 INFO:tasks.workunit.client.0.trial150.stderr:192899: waiting osd.5 seq 408021893122 2026-02-22T20:33:33.837 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:33:33.837 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893123 -lt 408021893122 2026-02-22T20:33:33.837 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:33.837 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953498 2026-02-22T20:33:33.837 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:33.837 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:33:33.837 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953498 2026-02-22T20:33:33.837 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:33.837 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953498 2026-02-22T20:33:33.837 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953498' 2026-02-22T20:33:35.476 INFO:tasks.workunit.client.0.trial150.stderr:192899: waiting osd.6 seq 1374389 2026-02-22T20:33:35.476 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:33:35.476 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953499 -lt 137438953499 2026-02-22T20:33:35.476 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:35.477 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789978 2026-02-22T20:33:35.477 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:35.477 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:33:35.477 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789978 2026-02-22T20:33:35.477 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:35.477 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789978 2026-02-22T20:33:35.477 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789978' 2026-02-22T20:33:35.477 INFO:tasks.workunit.client.0.trial150.stderr:192901: waiting osd.7 seq 158913789978 2026-02-22T20:33:35.477 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:33:35.477 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789978 -lt 158913789978 2026-02-22T20:33:35.477 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626457 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626457 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626457 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626457' 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: waiting osd.8 seq 180388626457 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626457 -lt 180388626457 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-403726925827 2026-02-22T20:33:35.478 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:35.479 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:33:35.479 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-403726925827 2026-02-22T20:33:35.479 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:35.479 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925827 2026-02-22T20:33:35.479 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 403726925827' 2026-02-22T20:33:35.479 INFO:tasks.workunit.client.0.trial150.stderr:192901: waiting osd.9 seq 403726925827 2026-02-22T20:33:35.479 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:35.479 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 403726925827 2026-02-22T20:33:35.479 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:33:35.489 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/53498 2026-02-22T20:33:35.490 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:33:35.490 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953499 -lt 137438953498 2026-02-22T20:33:35.490 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:35.490 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789977 2026-02-22T20:33:35.490 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:35.490 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:33:35.490 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789977 2026-02-22T20:33:35.490 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:35.490 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789977 2026-02-22T20:33:35.490 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789977' 2026-02-22T20:33:35.490 INFO:tasks.workunit.client.0.trial150.stderr:192899: waiting osd.7 seq 158913789977 2026-02-22T20:33:35.491 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:33:35.491 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789978 -lt 158913789977 2026-02-22T20:33:35.491 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:35.491 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626456 2026-02-22T20:33:35.491 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:35.491 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:33:35.491 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626456 2026-02-22T20:33:35.491 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:35.491 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626456 2026-02-22T20:33:35.491 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626456' 2026-02-22T20:33:35.491 INFO:tasks.workunit.client.0.trial150.stderr:192899: waiting osd.8 seq 180388626456 2026-02-22T20:33:35.491 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626457 -lt 180388626456 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-403726925826 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-403726925826 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925826 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 403726925826' 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: waiting osd.9 seq 403726925826 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 403726925826 2026-02-22T20:33:35.492 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:33:36.195 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standaceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:33:36.195 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:36.195 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925827 -lt 403726925827 2026-02-22T20:33:36.195 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:33:36.195 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:36.195 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:36.195 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:33:36.195 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:33:36.195 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:33:36.195 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:33:36.196 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:33:36.196 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:33:36.196 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:33:36.196 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:33:36.196 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:33:36.196 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:33:36.196 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:36.196 INFO:tasks.workunit.client.0.trial150.stderr:192901: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:36.196 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:33:36.196 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:33:36.196 INFO:tasks.workunit.client.0.trial150.stderr:192901: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:33:36.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:33:36.202 INFO:tasks.workunit.client.0.trial150.stderr:lone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:33:36.203 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:36.203 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925827 -lt 403726925826 2026-02-22T20:33:36.203 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:33:36.203 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:36.203 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:36.203 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:33:36.203 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:33:36.203 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:33:36.203 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:33:36.203 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:33:36.204 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:33:36.204 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:33:36.204 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:33:36.204 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:33:36.204 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:33:36.204 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:36.204 INFO:tasks.workunit.client.0.trial150.stderr:192899: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:36.204 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:33:36.204 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:33:36.204 INFO:tasks.workunit.client.0.trial150.stderr:192899: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:33:36.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:33:36.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:33:36.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 192897 2026-02-22T20:33:36.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:33:36.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:33:36.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:33:36.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-22T20:33:36.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:33:36.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:33:36.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:33:36.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2026-02-22T20:33:36.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-22T20:33:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-22T20:33:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=3 2026-02-22T20:33:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=5 2026-02-22T20:33:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-22T20:33:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:33:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:33:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:33:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 195769"' 2026-02-22T20:33:36.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 195769' 2026-02-22T20:33:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:33:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:33:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:33:36.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 195771"' 2026-02-22T20:33:36.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 195771' 2026-02-22T20:33:36.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:33:36.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-22T20:33:36.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 195769 195771' 2026-02-22T20:33:36.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:33:36.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:33:36.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 195769 2026-02-22T20:33:36.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/195772: /' 2026-02-22T20:33:36.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:33:36.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/195774: /' 2026-02-22T20:33:36.888 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:33:36.888 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:33:36.888 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:33:36.888 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:33:36.888 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:33:36.888 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:33:36.888 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:33:36.889 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:33:36.890 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:33:36.890 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:33:36.890 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-22T20:33:36.890 INFO:tasks.workunit.client.0.trial150.stderr:195772: remove 0#2:eb822e21:::SOMETHING:head# 2026-02-22T20:33:36.890 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:33:36.890 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:33:36.890 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:33:36.890 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:33:36.890 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:33:36.890 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:33:36.890 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:33:36.890 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:33:36.891 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: 195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:33:36.891 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:33:36.891 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-22T20:33:36.891 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:33:36.891 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:33:36.891 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:33:36.892 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:33:36.892 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-22T20:33:36.892 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:33:36.892 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-22T20:33:36.892 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:33:36.892 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:33:36.892 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:33:36.892 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:33:36.892 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:33:36.892 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:33:36.892 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:33:36.892 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:33:36.893 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:33:36.894 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:33:36.894 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:33:36.894 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:33:36.894 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:33:36.895 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:33:36.895 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:33:36.895 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:33:36.895 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:33:36.895 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:33:36.895 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:33:36.895 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:33:36.895 INFO:tasks.workunit.client.0.trial150.stderr:195772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:33:36.895 INFO:tasks.workunit.client.0.trial150.stderr:195772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:36.895 INFO:tasks.workunit.client.0.trial150.stderr:195772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:36.895 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:36.895 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:36.896 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:33:36.896 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:33:36.896 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:33:36.896 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:33:36.896 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:33:36.896 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:33:36.896 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:33:36.896 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:33:36.896 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:33:36.896 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:33:36.896 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:33:36.896 INFO:tasks.workunit.client.0.trial150.stderr:195772: start osd.3 2026-02-22T20:33:36.897 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:33:36.897 INFO:tasks.workunit.client.0.trial150.stderr:1activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:33:36.897 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:33:36.897 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-22T20:33:36.897 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-22T20:33:36.897 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:33:36.897 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:33:36.897 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:33:36.897 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:33:36.897 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:33:36.897 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:33:36.897 INFO:tasks.workunit.client.0.trial150.stderr:195774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:33:36.898 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:33:36.899 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-22T20:33:36.899 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-22T20:33:36.899 INFO:tasks.workunit.client.0.trial150.stderr:195774: start osd.5 2026-02-22T20:33:36.899 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:33:38.372 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:33:38.372 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:33:38.372 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:33:38.372 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:33:38.373 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:33:38.373 INFO:tasks.workunit.client.0.trial150.stderr:195772: 2026-02-22T20:33:36.896+0000 7f4a591fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:38.373 INFO:tasks.workunit.client.0.trial150.stderr:195772: 2026-02-22T20:33:36.896+0000 7f4a591fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:38.373 INFO:tasks.workunit.client.0.trial150.stderr:195772: 2026-02-22T20:33:36.896+0000 7f4a591fda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:38.373 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:33:38.373 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:33:38.373 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:33:38.373 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:33:38.373 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:33:38.373 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:38.373 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:33:38.373 INFO:tasks.workunit.client.0.trial150.stderr:195772: 0 2026-02-22T20:33:38.374 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:38.374 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:33:38.374 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:33:38.374 INFO:tasks.workunit.client.0.trial150.stderr:195772: 2026-02-22T20:33:37.340+0000 7f4a591fda40 -1 Falling back to public interface 2026-02-22T20:33:38.374 INFO:tasks.workunit.client.0.trial150.stderr:195772: 2026-02-22T20:33:37.964+0000 7f4a591fda40 -1 osd.3 96 log_to_monitors true 2026-02-22T20:33:38.374 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:33:38.374 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:38.374 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:33:38.374 INFO:tasks.workunit.client.0.trial150.stderr:195772: 1 2026-02-22T20:33:38.374 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:38.374 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:33:38.374 INFO:tasks.workunit.client.0.trial150.stderr:195772: osd.3 up in weight 1 up_from 100 up_thru 100 down_at 97 last_clean_interval [66,96) [v2:127.0.0.1:6826/1668549465,v1:127.0.0.1:6827/1668549465] [v2:127.0.0.1:6828/1668549465,v1:127.0.0.1:6829/1668549465] exists,up 13ed2ee7-2de8-444f-8bda-cc72d609f742 2026-02-22T20:33:38.375 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:33:38.375 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:33:38.375 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:33:38.375 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:33:38.375 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:33:38.375 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:33:38.375 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:33:38.375 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:33:38.375 INFO:tasks.workunit.client.0.trial150.stderr:195772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:33:38.383 INFO:tasks.workunit.client.0.trial150.stderr:195772: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593:95774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-22T20:33:38.383 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-22T20:33:38.384 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:33:38.384 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:33:38.384 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:33:38.384 INFO:tasks.workunit.client.0.trial150.stderr:195774: 2026-02-22T20:33:36.896+0000 7fe24cc57a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:38.384 INFO:tasks.workunit.client.0.trial150.stderr:195774: 2026-02-22T20:33:36.896+0000 7fe24cc57a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:38.384 INFO:tasks.workunit.client.0.trial150.stderr:195774: 2026-02-22T20:33:36.896+0000 7fe24cc57a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:38.384 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-22T20:33:38.384 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:33:38.384 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-22T20:33:38.384 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:33:38.384 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:33:38.384 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:38.385 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:33:38.385 INFO:tasks.workunit.client.0.trial150.stderr:195774: 0 2026-02-22T20:33:38.385 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:38.385 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:33:38.385 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:33:38.385 INFO:tasks.workunit.client.0.trial150.stderr:195774: 2026-02-22T20:33:37.348+0000 7fe24cc57a40 -1 Falling back to public interface 2026-02-22T20:33:38.385 INFO:tasks.workunit.client.0.trial150.stderr:195774: 2026-02-22T20:33:37.996+0000 7fe24cc57a40 -1 osd.5 96 log_to_monitors true 2026-02-22T20:33:38.385 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:33:38.385 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:38.385 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: 1 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: osd.5 up in weight 1 up_from 100 up_thru 100 down_at 97 last_clean_interval [95,96) [v2:127.0.0.1:6874/3454858066,v1:127.0.0.1:6875/3454858066] [v2:127.0.0.1:6876/3454858066,v1:127.0.0.1:6877/3454858066] exists,up b815c525-94b6-4f0c-bea0-1363d5e04434 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:33:38.386 INFO:tasks.workunit.client.0.trial150.stderr:195774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:33:38.931 INFO:tasks.workunit.client.0.trial150.stderr:195774: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:33:38.931 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:33:38.931 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:33:38.931 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:33:38.931 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:33:38.931 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:33:38.931 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:33:38.932 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:33:38.932 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:33:38.932 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:33:38.932 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:33:38.932 INFO:tasks.workunit.client.0.trial150.stderr:195772: 1 2026-02-22T20:33:38.932 INFO:tasks.workunit.client.0.trial150.stderr:195772: 2 2026-02-22T20:33:38.932 INFO:tasks.workunit.client.0.trial150.stderr:195772: 3 2026-02-22T20:33:38.932 INFO:tasks.workunit.client.0.trial150.stderr:195772: 4 2026-02-22T20:33:38.932 INFO:tasks.workunit.client.0.trial150.stderr:195772: 5 2026-02-22T20:33:38.932 INFO:tasks.workunit.client.0.trial150.stderr:195772: 6 2026-02-22T20:33:38.932 INFO:tasks.workunit.client.0.trial150.stderr:195772: 7 2026-02-22T20:33:38.932 INFO:tasks.workunit.client.0.trial150.stderr:195772: 8 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: 9' 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836513 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836513 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513' 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672993 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672993 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672993' 2026-02-22T20:33:38.933 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:38.934 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:33:38.934 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542176 2026-02-22T20:33:38.934 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542176 2026-02-22T20:33:38.934 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542176' 2026-02-22T20:33:38.934 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:38.934 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:33:38.934 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=429496729602 2026-02-22T20:33:38.934 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 429496729602 2026-02-22T20:33:38.934 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542176 3-429496729602' 2026-02-22T20:33:38.934 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:38.936 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:225 get_timeout_delays: echo true 2026-02-22T20:33:38.936 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:33:38.936 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:33:38.936 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:33:38.936 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:33:38.936 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:33:38.936 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:33:38.936 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:33:38.936 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:33:38.936 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: 1 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: 2 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: 3 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: 4 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: 5 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: 6 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: 7 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: 8 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: 9' 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:38.937 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836514 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836514 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514' 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672994 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672994 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672994' 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542177 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542177 2026-02-22T20:33:38.938 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672994 2-60129542177' 2026-02-22T20:33:38.939 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:38.939 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:33:38.939 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=429496729603 2026-02-22T20:33:38.939 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 429496729603 2026-02-22T20:33:38.939 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672994 2-60129542177 3-429496729603' 2026-02-22T20:33:38.939 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:39.368 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215134 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215134 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542176 3-429496729602 4-103079215134' 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=429496729602 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 429496729602 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542176 3-429496729602 4-103079215134 5-429496729602' 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953501 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953501 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542176 3-429496729602 4-103079215134 5-429496729602 6-137438953501' 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789980 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789980 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542176 3-429496729602 4-103079215134 5-429496729602 6-137438953501 7-158913789980' 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626460 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626460 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542176 3-429496729602 4-103079215134 5-429496729602 6-137438953501 7-158913789980 8-180388626460' 2026-02-22T20:33:39.369 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:39.370 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:33:39.370 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925829 2026-02-22T20:33:39.370 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925829 2026-02-22T20:33:39.370 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672993 2-60129542176 3-429496729602 4-103079215134 5-429496729602 6-137438953501 7-158913789980 8-180388626460 9-403726925829' 2026-02-22T20:33:39.370 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/8: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215135 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215135 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672994 2-60129542177 3-429496729603 4-103079215135' 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=429496729603 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 429496729603 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672994 2-60129542177 3-429496729603 4-103079215135 5-429496729603' 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953502 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953502 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672994 2-60129542177 3-429496729603 4-103079215135 5-429496729603 6-137438953502' 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789981 2026-02-22T20:33:39.372 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789981 2026-02-22T20:33:39.373 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672994 2-60129542177 3-429496729603 4-103079215135 5-429496729603 6-137438953502 7-158913789981' 2026-02-22T20:33:39.373 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:39.373 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:33:39.373 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626461 2026-02-22T20:33:39.373 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626461 2026-02-22T20:33:39.373 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672994 2-60129542177 3-429496729603 4-103079215135 5-429496729603 6-137438953502 7-158913789981 8-180388626461' 2026-02-22T20:33:39.373 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:39.373 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:33:39.373 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925830 2026-02-22T20:33:39.373 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925830 2026-02-22T20:33:39.373 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672994 2-60129542177 3-429496729603 4-103079215135 5-429496729603 6-137438953502 7-158913789981 8-180388626461 9-403726925830' 2026-02-22T20:33:39.373 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:41.018 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836513 2026-02-22T20:33:41.018 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.018 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:33:41.019 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836513 2026-02-22T20:33:41.019 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:41.019 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836513 2026-02-22T20:33:41.019 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2026-02-22T20:33:41.019 INFO:tasks.workunit.client.0.trial150.stderr:195772: waiting osd.0 seq 21474836513 2026-02-22T20:33:41.020 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:33:41.020 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836514 -lt 21474836513 2026-02-22T20:33:41.020 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:41.020 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672993 2026-02-22T20:33:41.020 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.020 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:33:41.021 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672993 2026-02-22T20:33:41.021 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:41.021 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672993 2026-02-22T20:33:41.021 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2026-02-22T20:33:41.021 INFO:tasks.workunit.client.0.trial150.stderr:195772: waiting osd.1 seq 42949672993 2026-02-22T20:33:41.021 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:33:41.022 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672994 -lt 42949672993 2026-02-22T20:33:41.022 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:41.022 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542176 2026-02-22T20:33:41.022 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.022 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:33:41.022 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542176 2026-02-22T20:33:41.022 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:41.023 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542176 2026-02-22T20:33:41.023 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542176' 2026-02-22T20:33:41.023 INFO:tasks.workunit.client.0.trial150.stderr:195772: waiting osd.2 seq 60129542176 2026-02-22T20:33:41.023 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:33:41.023 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542175 -lt 60129542176 2026-02-22T20:33:41.023 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:33:41.024 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:33:41.024 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:33:41.024 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542177 -lt 60129542176 2026-02-22T20:33:41.026 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: fluubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836514 2026-02-22T20:33:41.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:33:41.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836514 2026-02-22T20:33:41.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:41.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836514 2026-02-22T20:33:41.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2026-02-22T20:33:41.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: waiting osd.0 seq 21474836514 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836514 -lt 21474836514 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672994 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672994 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672994 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: waiting osd.1 seq 42949672994 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:33:41.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672994 -lt 42949672994 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542177 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542177 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542177 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542177' 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: waiting osd.2 seq 60129542177 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542175 -lt 60129542177 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:33:41.029 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:33:41.030 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542177 -lt 60129542177 2026-02-22T20:33:41.517 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:41.517 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-429496729602 2026-02-22T20:33:41.517 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.517 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:33:41.517 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-429496729602 2026-02-22T20:33:41.518 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:41.518 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=429496729602 2026-02-22T20:33:41.518 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 429496729602' 2026-02-22T20:33:41.518 INFO:tasks.workunit.client.0.trial150.stderr:195772: waiting osd.3 seq 429496729602 2026-02-22T20:33:41.518 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:33:41.518 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729603 -lt 429496729602 2026-02-22T20:33:41.518 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:41.518 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215134 2026-02-22T20:33:41.518 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.518 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:33:41.518 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215134 2026-02-22T20:33:41.518 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:41.519 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215134 2026-02-22T20:33:41.519 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215134' 2026-02-22T20:33:41.519 INFO:tasks.workunit.client.0.trial150.stderr:195772: waiting osd.4 seq 103079215134 2026-02-22T20:33:41.519 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:33:41.519 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215136 -lt 103079215134 2026-02-22T20:33:41.519 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:41.519 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-429496729602 2026-02-22T20:33:41.519 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.519 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:33:41.519 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-429496729602 2026-02-22T20:33:41.519 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:41.519 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=429496729602 2026-02-22T20:33:41.520 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 429496729602' 2026-02-22T20:33:41.520 INFO:tasks.workunit.client.0.trial150.stderr:195772: waiting osd.5 seq 429496729602 2026-02-22T20:33:41.520 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:33:41.520 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729603 -lt 429496729602 2026-02-22T20:33:41.520 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:41.520 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953501 2026-02-22T20:33:41.520 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.520 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:33:41.528 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qsh_pg_stats: for s in $seqs 2026-02-22T20:33:41.528 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-429496729603 2026-02-22T20:33:41.528 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-429496729603 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=429496729603 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 429496729603' 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: waiting osd.3 seq 429496729603 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729603 -lt 429496729603 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215135 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:33:41.529 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215135 2026-02-22T20:33:41.530 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:41.530 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215135 2026-02-22T20:33:41.530 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215135' 2026-02-22T20:33:41.530 INFO:tasks.workunit.client.0.trial150.stderr:195774: waiting osd.4 seq 103079215135 2026-02-22T20:33:41.530 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:33:41.530 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215136 -lt 103079215135 2026-02-22T20:33:41.530 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:41.530 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-429496729603 2026-02-22T20:33:41.530 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.530 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:33:41.530 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-429496729603 2026-02-22T20:33:41.530 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:41.531 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=429496729603 2026-02-22T20:33:41.531 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 429496729603' 2026-02-22T20:33:41.531 INFO:tasks.workunit.client.0.trial150.stderr:195774: waiting osd.5 seq 429496729603 2026-02-22T20:33:41.531 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:33:41.531 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 429496729603 -lt 429496729603 2026-02-22T20:33:41.531 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:41.531 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953502 2026-02-22T20:33:41.531 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:41.531 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:33:42.001 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953501 2026-02-22T20:33:42.001 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:42.001 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953501 2026-02-22T20:33:42.001 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953501' 2026-02-22T20:33:42.001 INFO:tasks.workunit.client.0.trial150.stderr:195772: waiting osd.6 seq 137438953501 2026-02-22T20:33:42.001 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:33:42.002 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953502 -lt 137438953501 2026-02-22T20:33:42.002 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:42.002 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789980 2026-02-22T20:33:42.002 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:42.002 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:33:42.002 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789980 2026-02-22T20:33:42.002 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:42.002 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789980 2026-02-22T20:33:42.002 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789980' 2026-02-22T20:33:42.002 INFO:tasks.workunit.client.0.trial150.stderr:195772: waiting osd.7 seq 158913789980 2026-02-22T20:33:42.002 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:33:42.002 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789982 -lt 158913789980 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626460 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626460 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626460 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626460' 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: waiting osd.8 seq 180388626460 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626461 -lt 180388626460 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:42.003 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-403726925829 2026-02-22T20:33:42.004 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:42.004 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:33:42.004 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-403726925829 2026-02-22T20:33:42.004 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:42.004 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925829 2026-02-22T20:33:42.025 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting a/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953502 2026-02-22T20:33:42.025 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:42.025 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953502 2026-02-22T20:33:42.025 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953502' 2026-02-22T20:33:42.025 INFO:tasks.workunit.client.0.trial150.stderr:195774: waiting osd.6 seq 137438953502 2026-02-22T20:33:42.025 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:33:42.026 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953502 -lt 137438953502 2026-02-22T20:33:42.026 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:42.026 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789981 2026-02-22T20:33:42.026 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:42.026 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:33:42.026 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789981 2026-02-22T20:33:42.026 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:42.026 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789981 2026-02-22T20:33:42.026 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789981' 2026-02-22T20:33:42.026 INFO:tasks.workunit.client.0.trial150.stderr:195774: waiting osd.7 seq 158913789981 2026-02-22T20:33:42.026 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:33:42.026 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789982 -lt 158913789981 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626461 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626461 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626461 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626461' 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: waiting osd.8 seq 180388626461 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626461 -lt 180388626461 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:42.027 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-403726925830 2026-02-22T20:33:42.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:42.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:33:42.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-403726925830 2026-02-22T20:33:42.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:42.028 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925830 2026-02-22T20:33:42.715 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 403726925829' 2026-02-22T20:33:42.715 INFO:tasks.workunit.client.0.trial150.stderr:195772: waiting osd.9 seq 403726925829 2026-02-22T20:33:42.716 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:42.716 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925830 -lt 403726925829 2026-02-22T20:33:42.716 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:33:42.716 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:42.716 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:42.716 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:33:42.716 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:33:42.716 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:33:42.716 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:33:42.716 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:33:42.716 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:33:42.717 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:33:42.717 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:33:42.717 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:33:42.717 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:33:42.717 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:42.717 INFO:tasks.workunit.client.0.trial150.stderr:195772: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:42.717 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:33:42.717 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:33:42.717 INFO:tasks.workunit.client.0.trial150.stderr:195772: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:33:42.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:33:42.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:33:42.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 195771 2026-02-22T20:33:42.721 INFO:tasks.workunit.client.0.trial150.stderr:osd.9 seq 403726925830' 2026-02-22T20:33:42.721 INFO:tasks.workunit.client.0.trial150.stderr:195774: waiting osd.9 seq 403726925830 2026-02-22T20:33:42.722 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:42.722 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925830 -lt 403726925830 2026-02-22T20:33:42.722 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:33:42.722 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:42.722 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:42.722 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:33:42.722 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:33:42.722 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:33:42.722 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:33:42.722 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:33:42.722 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:33:42.723 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:33:42.723 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:33:42.723 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:33:42.723 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:33:42.723 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:42.723 INFO:tasks.workunit.client.0.trial150.stderr:195774: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:42.723 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:33:42.723 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:33:42.723 INFO:tasks.workunit.client.0.trial150.stderr:195774: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:33:42.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:33:42.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:33:42.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:33:42.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:33:42.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-22T20:33:42.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:33:42.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-22T20:33:42.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:33:42.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:33:42.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:33:42.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:33:42.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-22T20:33:42.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-22T20:33:42.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:33:42.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:33:42.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:42.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:42.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:42.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:43.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:33:23.690461+0000 2026-02-22T20:33:43.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:33:43.181 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:33:43.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:33:23.690461+0000 2026-02-22T20:33:43.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:33:43.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:33:23.690461+0000 2026-02-22T20:33:43.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:33:43.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:33:43.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:43.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:43.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:43.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:43.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:43.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:43.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:23.690461+0000 '>' 2026-02-22T20:33:23.690461+0000 2026-02-22T20:33:43.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:44.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:44.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:44.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:44.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:44.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:44.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:44.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:44.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:23.690461+0000 '>' 2026-02-22T20:33:23.690461+0000 2026-02-22T20:33:44.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:45.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:45.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:45.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:45.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:45.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:45.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:45.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:45.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:23.690461+0000 '>' 2026-02-22T20:33:23.690461+0000 2026-02-22T20:33:45.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:46.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:46.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:46.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:46.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:46.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:46.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:46.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:46.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:23.690461+0000 '>' 2026-02-22T20:33:23.690461+0000 2026-02-22T20:33:46.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:47.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:47.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:47.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:47.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:47.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:47.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:47.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:47.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:23.690461+0000 '>' 2026-02-22T20:33:23.690461+0000 2026-02-22T20:33:47.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:33:48.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:33:48.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:33:48.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:33:48.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:33:48.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:33:48.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:33:48.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:33:49.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:33:43.906536+0000 '>' 2026-02-22T20:33:23.690461+0000 2026-02-22T20:33:49.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:33:49.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-22T20:33:49.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:33:49.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:33:49.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:33:49.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 199206"' 2026-02-22T20:33:49.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 199206' 2026-02-22T20:33:49.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:33:49.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:33:49.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:33:49.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 199208"' 2026-02-22T20:33:49.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 199208' 2026-02-22T20:33:49.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:33:49.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-22T20:33:49.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 199206 199208' 2026-02-22T20:33:49.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:33:49.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:33:49.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 199206 2026-02-22T20:33:49.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/199209: /' 2026-02-22T20:33:49.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:33:49.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/199211: /' 2026-02-22T20:33:49.646 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:33:49.647 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:33:49.647 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-22T20:33:49.647 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:33:49.647 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:33:49.647 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:33:49.647 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:33:49.647 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-22T20:33:49.647 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:33:49.647 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-22T20:33:49.647 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2026-02-22T20:33:49.648 INFO:tasks.workunit.client.0.trial150.stderr:199211: _ 2026-02-22T20:33:49.649 INFO:tasks.workunit.client.0.trial150.stderr:199211: hinfo_key 2026-02-22T20:33:49.649 INFO:tasks.workunit.client.0.trial150.stderr:199211: snapset 2026-02-22T20:33:49.649 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-22T20:33:49.649 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:33:49.649 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:33:49.649 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-22T20:33:49.649 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:33:49.649 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:33:49.649 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:33:49.649 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:33:49.650 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:33:49.650 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:33:49.650 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-22T20:33:49.650 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-22T20:33:49.650 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:33:49.650 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:33:49.650 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:33:49.651 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:33:49.651 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:33:49.651 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:33:49.651 INFO:tasks.workunit.client.0.trial150.stderr:199211: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:33:49.651 INFO:tasks.workunit.client.0.trial150.stderr:199211: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:49.651 INFO:tasks.workunit.client.0.trial150.stderr:199211: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:49.651 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:49.651 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:49.651 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:33:49.651 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:33:49.651 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:33:49.651 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:33:49.652 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:33:49.652 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:33:49.652 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:33:49.652 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:33:49.652 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:33:49.652 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-22T20:33:49.652 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-22T20:33:49.652 INFO:tasks.workunit.client.0.trial150.stderr:199211: start osd.5 2026-02-22T20:33:49.670 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:33:49.671 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:33:49.671 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:33:49.671 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:33:49.671 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:33:49.671 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:33:49.671 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:33:49.671 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:33:49.671 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:33:49.671 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:33:49.671 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:33:49.672 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:33:49.672 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:33:49.672 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:33:49.672 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:33:49.672 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:33:49.672 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:33:49.672 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:33:49.672 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:33:49.672 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:33:49.672 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:33:49.672 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:33:49.672 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-22T20:33:49.673 INFO:tasks.workunit.client.0.trial150.stderr:199209: Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-22T20:33:49.673 INFO:tasks.workunit.client.0.trial150.stderr:199209: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-22T20:33:49.673 INFO:tasks.workunit.client.0.trial150.stderr:199209: _ 2026-02-22T20:33:49.673 INFO:tasks.workunit.client.0.trial150.stderr:199209: hinfo_key 2026-02-22T20:33:49.673 INFO:tasks.workunit.client.0.trial150.stderr:199209: snapset 2026-02-22T20:33:49.673 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:33:49.673 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:33:49.673 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:33:49.673 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:33:49.673 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:33:49.673 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:33:49.673 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:33:49.674 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:33:49.674 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:33:49.674 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:33:49.674 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:33:49.674 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:33:49.674 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:33:49.675 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:33:49.676 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:33:49.676 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:33:49.676 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:33:49.676 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:33:49.676 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:33:49.676 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:33:49.676 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:33:49.676 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:33:49.676 INFO:tasks.workunit.client.0.trial150.stderr:199209: start osd.3 2026-02-22T20:33:52.286 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.very_ops 2026-02-22T20:33:52.286 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-22T20:33:52.286 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-22T20:33:52.286 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:33:52.287 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:33:52.287 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:33:52.287 INFO:tasks.workunit.client.0.trial150.stderr:199211: 2026-02-22T20:33:49.659+0000 7f5a51596a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:52.287 INFO:tasks.workunit.client.0.trial150.stderr:199211: 2026-02-22T20:33:49.659+0000 7f5a51596a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:52.287 INFO:tasks.workunit.client.0.trial150.stderr:199211: 2026-02-22T20:33:49.659+0000 7f5a51596a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:52.287 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-22T20:33:52.287 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:33:52.287 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-22T20:33:52.287 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:33:52.287 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:33:52.287 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:52.287 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:33:52.288 INFO:tasks.workunit.client.0.trial150.stderr:199211: 0 2026-02-22T20:33:52.288 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:52.288 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:33:52.288 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:33:52.288 INFO:tasks.workunit.client.0.trial150.stderr:199211: 2026-02-22T20:33:50.199+0000 7f5a51596a40 -1 Falling back to public interface 2026-02-22T20:33:52.288 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:33:52.288 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:52.288 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:33:52.288 INFO:tasks.workunit.client.0.trial150.stderr:199211: 1 2026-02-22T20:33:52.288 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: 2026-02-22T20:33:51.003+0000 7f5a51596a40 -1 osd.5 101 log_to_monitors true 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: 2 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: osd.5 up in weight 1 up_from 105 up_thru 105 down_at 102 last_clean_interval [100,101) [v2:127.0.0.1:6826/4089024597,v1:127.0.0.1:6828/4089024597] [v2:127.0.0.1:6830/4089024597,v1:127.0.0.1:6832/4089024597] exists,up b815c525-94b6-4f0c-bea0-1363d5e04434 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:33:52.289 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:33:52.290 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:33:52.311 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntulog' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:33:52.311 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:33:52.311 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: 2026-02-22T20:33:49.679+0000 7f96fae75a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: 2026-02-22T20:33:49.679+0000 7f96fae75a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: 2026-02-22T20:33:49.679+0000 7f96fae75a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:52.312 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: 0 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: 2026-02-22T20:33:50.199+0000 7f96fae75a40 -1 Falling back to public interface 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: 1 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: 2026-02-22T20:33:51.019+0000 7f96fae75a40 -1 osd.3 101 log_to_monitors true 2026-02-22T20:33:52.313 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:33:52.314 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:33:52.314 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:52.314 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:33:52.314 INFO:tasks.workunit.client.0.trial150.stderr:199209: 2 2026-02-22T20:33:52.314 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:52.314 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:33:52.314 INFO:tasks.workunit.client.0.trial150.stderr:199209: osd.3 up in weight 1 up_from 105 up_thru 105 down_at 102 last_clean_interval [100,101) [v2:127.0.0.1:6827/1593778321,v1:127.0.0.1:6829/1593778321] [v2:127.0.0.1:6831/1593778321,v1:127.0.0.1:6833/1593778321] exists,up 13ed2ee7-2de8-444f-8bda-cc72d609f742 2026-02-22T20:33:52.314 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:33:52.314 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:33:52.764 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/cl/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:33:52.764 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:33:52.764 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:33:52.764 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:33:52.764 INFO:tasks.workunit.client.0.trial150.stderr:199211: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:33:52.764 INFO:tasks.workunit.client.0.trial150.stderr:199211: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:33:52.764 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:33:52.764 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:33:52.764 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:33:52.764 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:33:52.764 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:33:52.764 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: 1 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: 2 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: 3 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: 4 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: 5 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: 6 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: 7 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: 8 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: 9' 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836518 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836518 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518' 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:52.765 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:33:52.766 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672997 2026-02-22T20:33:52.766 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672997 2026-02-22T20:33:52.766 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997' 2026-02-22T20:33:52.766 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:52.766 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:33:52.766 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542181 2026-02-22T20:33:52.766 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542181 2026-02-22T20:33:52.766 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-60129542181' 2026-02-22T20:33:52.766 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standaloone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:33:52.796 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: 1 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: 2 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: 3 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: 4 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: 5 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: 6 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: 7 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: 8 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: 9' 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836519 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836519 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519' 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:33:52.797 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672998 2026-02-22T20:33:52.798 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672998 2026-02-22T20:33:52.798 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998' 2026-02-22T20:33:52.798 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:52.798 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:33:52.798 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542182 2026-02-22T20:33:52.798 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542182 2026-02-22T20:33:53.197 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474ne/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:33:53.197 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=450971566082 2026-02-22T20:33:53.197 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 450971566082 2026-02-22T20:33:53.197 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-60129542181 3-450971566082' 2026-02-22T20:33:53.198 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.198 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:33:53.198 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215139 2026-02-22T20:33:53.198 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215139 2026-02-22T20:33:53.198 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-60129542181 3-450971566082 4-103079215139' 2026-02-22T20:33:53.198 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.198 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:33:53.198 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=450971566082 2026-02-22T20:33:53.198 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 450971566082 2026-02-22T20:33:53.198 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-60129542181 3-450971566082 4-103079215139 5-450971566082' 2026-02-22T20:33:53.198 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.199 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:33:53.199 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953506 2026-02-22T20:33:53.199 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953506 2026-02-22T20:33:53.199 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-60129542181 3-450971566082 4-103079215139 5-450971566082 6-137438953506' 2026-02-22T20:33:53.199 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.199 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:33:53.199 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789985 2026-02-22T20:33:53.199 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789985 2026-02-22T20:33:53.199 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-60129542181 3-450971566082 4-103079215139 5-450971566082 6-137438953506 7-158913789985' 2026-02-22T20:33:53.199 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.199 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:33:53.199 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626464 2026-02-22T20:33:53.200 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626464 2026-02-22T20:33:53.200 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-60129542181 3-450971566082 4-103079215139 5-450971566082 6-137438953506 7-158913789985 8-180388626464' 2026-02-22T20:33:53.200 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.229 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: 836519 1-42949672998 2-60129542182' 2026-02-22T20:33:53.229 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.230 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:33:53.230 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=450971566083 2026-02-22T20:33:53.230 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 450971566083 2026-02-22T20:33:53.230 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542182 3-450971566083' 2026-02-22T20:33:53.230 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.230 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:33:53.230 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215140 2026-02-22T20:33:53.230 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215140 2026-02-22T20:33:53.230 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542182 3-450971566083 4-103079215140' 2026-02-22T20:33:53.230 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.230 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:33:53.230 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=450971566083 2026-02-22T20:33:53.231 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 450971566083 2026-02-22T20:33:53.231 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542182 3-450971566083 4-103079215140 5-450971566083' 2026-02-22T20:33:53.231 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.231 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:33:53.231 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=137438953507 2026-02-22T20:33:53.231 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 137438953507 2026-02-22T20:33:53.231 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542182 3-450971566083 4-103079215140 5-450971566083 6-137438953507' 2026-02-22T20:33:53.231 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.231 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:33:53.231 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789986 2026-02-22T20:33:53.231 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789986 2026-02-22T20:33:53.231 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542182 3-450971566083 4-103079215140 5-450971566083 6-137438953507 7-158913789986' 2026-02-22T20:33:53.232 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.232 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:33:53.232 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626465 2026-02-22T20:33:53.232 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626465 2026-02-22T20:33:53.636 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542182 3-450971566083 4-103079215140 5-450971566083 6-137438953507 7-158913789986 8-180388626465' 2026-02-22T20:33:53.636 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:33:53.636 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:33:53.636 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925835 2026-02-22T20:33:53.636 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925835 2026-02-22T20:33:53.637 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-60129542182 3-450971566083 4-103079215140 5-450971566083 6-137438953507 7-158913789986 8-180388626465 9-403726925835' 2026-02-22T20:33:53.637 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:53.637 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836519 2026-02-22T20:33:53.637 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:53.637 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:33:53.637 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836519 2026-02-22T20:33:53.637 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:53.637 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836519 2026-02-22T20:33:53.637 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2026-02-22T20:33:53.637 INFO:tasks.workunit.client.0.trial150.stderr:199209: waiting osd.0 seq 21474836519 2026-02-22T20:33:53.637 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836519 -lt 21474836519 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672998 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672998 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672998 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: waiting osd.1 seq 42949672998 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672998 -lt 42949672998 2026-02-22T20:33:53.638 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:53.639 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542182 2026-02-22T20:33:53.639 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:53.639 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:33:53.639 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542182 2026-02-22T20:33:53.639 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:53.639 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542182 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 sflush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=403726925834 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 403726925834 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-60129542181 3-450971566082 4-103079215139 5-450971566082 6-137438953506 7-158913789985 8-180388626464 9-403726925834' 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836518 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836518 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836518 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: waiting osd.0 seq 21474836518 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836519 -lt 21474836518 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672997 2026-02-22T20:33:53.752 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672997 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672997 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: waiting osd.1 seq 42949672997 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672998 -lt 42949672997 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542181 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542181 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542181 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542181' 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: waiting osd.2 seq 60129542181 2026-02-22T20:33:53.753 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:33:55.236 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542182 -lt 60129542181 2026-02-22T20:33:55.236 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.236 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-450971566082 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-450971566082 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=450971566082 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 450971566082' 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: waiting osd.3 seq 450971566082 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 450971566082 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566083 -lt 450971566082 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215139 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215139 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215139 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215139' 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: waiting osd.4 seq 103079215139 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215141 -lt 103079215139 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-450971566082 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-450971566082 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=450971566082 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 450971566082' 2026-02-22T20:33:55.237 INFO:tasks.workunit.client.0.trial150.stderr:199211: waiting osd.5 seq 450971566082 2026-02-22T20:33:55.299 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: eq 60129542182' 2026-02-22T20:33:55.299 INFO:tasks.workunit.client.0.trial150.stderr:199209: waiting osd.2 seq 60129542182 2026-02-22T20:33:55.300 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:33:55.300 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542182 -lt 60129542182 2026-02-22T20:33:55.300 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.300 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-450971566083 2026-02-22T20:33:55.300 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:55.300 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:33:55.300 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-450971566083 2026-02-22T20:33:55.300 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:55.300 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=450971566083 2026-02-22T20:33:55.300 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 450971566083' 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: waiting osd.3 seq 450971566083 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 450971566083 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566083 -lt 450971566083 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-103079215140 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-103079215140 2026-02-22T20:33:55.301 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:55.302 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215140 2026-02-22T20:33:55.302 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 103079215140' 2026-02-22T20:33:55.302 INFO:tasks.workunit.client.0.trial150.stderr:199209: waiting osd.4 seq 103079215140 2026-02-22T20:33:55.302 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:33:55.302 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215141 -lt 103079215140 2026-02-22T20:33:55.302 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.302 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-450971566083 2026-02-22T20:33:55.302 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:55.302 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:33:55.302 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-450971566083 2026-02-22T20:33:55.302 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:55.302 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=450971566083 2026-02-22T20:33:55.897 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helceph osd last-stat-seq 5 2026-02-22T20:33:55.897 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566083 -lt 450971566082 2026-02-22T20:33:55.897 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.897 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953506 2026-02-22T20:33:55.897 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:55.897 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:33:55.898 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953506 2026-02-22T20:33:55.898 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:55.898 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953506 2026-02-22T20:33:55.898 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953506' 2026-02-22T20:33:55.898 INFO:tasks.workunit.client.0.trial150.stderr:199211: waiting osd.6 seq 137438953506 2026-02-22T20:33:55.898 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:33:55.898 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953507 -lt 137438953506 2026-02-22T20:33:55.898 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.898 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789985 2026-02-22T20:33:55.898 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:55.898 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:33:55.898 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789985 2026-02-22T20:33:55.899 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:55.899 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789985 2026-02-22T20:33:55.899 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789985' 2026-02-22T20:33:55.899 INFO:tasks.workunit.client.0.trial150.stderr:199211: waiting osd.7 seq 158913789985 2026-02-22T20:33:55.899 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:33:55.899 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789986 -lt 158913789985 2026-02-22T20:33:55.899 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.899 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626464 2026-02-22T20:33:55.899 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:55.899 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:33:55.899 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626464 2026-02-22T20:33:55.899 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:55.900 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626464 2026-02-22T20:33:55.900 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626464' 2026-02-22T20:33:55.900 INFO:tasks.workunit.client.0.trial150.stderr:199211: waiting osd.8 seq 180388626464 2026-02-22T20:33:55.900 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:33:55.900 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626466 -lt 180388626464 2026-02-22T20:33:55.900 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.900 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-403726925834 2026-02-22T20:33:55.953 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/pers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 450971566083' 2026-02-22T20:33:55.953 INFO:tasks.workunit.client.0.trial150.stderr:199209: waiting osd.5 seq 450971566083 2026-02-22T20:33:55.953 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:33:55.953 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566083 -lt 450971566083 2026-02-22T20:33:55.953 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.953 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-137438953507 2026-02-22T20:33:55.953 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:55.953 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:33:55.954 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-137438953507 2026-02-22T20:33:55.954 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:55.954 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=137438953507 2026-02-22T20:33:55.954 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 137438953507' 2026-02-22T20:33:55.954 INFO:tasks.workunit.client.0.trial150.stderr:199209: waiting osd.6 seq 137438953507 2026-02-22T20:33:55.954 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:33:55.954 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 137438953507 -lt 137438953507 2026-02-22T20:33:55.954 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.954 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-158913789986 2026-02-22T20:33:55.954 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:55.954 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:33:55.954 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-158913789986 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789986 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 158913789986' 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: waiting osd.7 seq 158913789986 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789986 -lt 158913789986 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-180388626465 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-180388626465 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:55.955 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626465 2026-02-22T20:33:55.956 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 180388626465' 2026-02-22T20:33:55.956 INFO:tasks.workunit.client.0.trial150.stderr:199209: waiting osd.8 seq 180388626465 2026-02-22T20:33:55.956 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:33:55.956 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626466 -lt 180388626465 2026-02-22T20:33:56.611 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qacephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:56.611 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:33:56.611 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-403726925834 2026-02-22T20:33:56.611 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:56.611 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925834 2026-02-22T20:33:56.611 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 403726925834' 2026-02-22T20:33:56.611 INFO:tasks.workunit.client.0.trial150.stderr:199211: waiting osd.9 seq 403726925834 2026-02-22T20:33:56.611 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:56.611 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925835 -lt 403726925834 2026-02-22T20:33:56.612 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:33:56.612 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:56.612 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:56.612 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:33:56.612 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:33:56.612 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:33:56.612 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:33:56.612 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:33:56.612 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:33:56.612 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:33:56.612 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:33:56.612 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:33:56.613 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:33:56.613 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:56.613 INFO:tasks.workunit.client.0.trial150.stderr:199211: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:56.613 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:33:56.613 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:33:56.613 INFO:tasks.workunit.client.0.trial150.stderr:199211: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:33:56.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:33:56.668 INFO:tasks.workunit.client.0.trial150.stderr:/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:33:56.668 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-403726925835 2026-02-22T20:33:56.668 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:33:56.668 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:33:56.668 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-403726925835 2026-02-22T20:33:56.668 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=403726925835 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 403726925835' 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: waiting osd.9 seq 403726925835 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 403726925835 -lt 403726925835 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:33:56.669 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:33:56.670 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:33:56.670 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:33:56.670 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:33:56.670 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:33:56.670 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:33:56.670 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:33:56.670 INFO:tasks.workunit.client.0.trial150.stderr:199209: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:33:56.670 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:33:56.670 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:33:56.670 INFO:tasks.workunit.client.0.trial150.stderr:199209: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:33:56.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:33:56.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:33:56.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 199208 2026-02-22T20:33:56.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:33:56.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:33:56.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:33:56.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-22T20:33:56.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:33:56.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:33:56.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:33:56.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:33:56.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:33:56.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:33:56.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:33:56.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:33:56.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:33:56.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:33:56.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:33:56.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:33:57.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:33:57.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:33:57.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:33:57.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:33:57.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:33:57.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:33:57.009 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:33:57.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:33:57.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:33:57.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:33:57.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:33:57.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:33:57.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:33:57.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:33:57.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:33:57.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:57.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:57.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:33:57.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:33:57.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:33:57.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:33:57.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:33:57.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:33:57.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:33:57.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:33:57.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:33:57.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:33:57.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:33:57.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:33:57.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:33:57.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:33:57.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:33:57.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:33:57.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:33:57.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:33:57.057 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:33:57.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:33:57.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:33:57.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:33:57.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:33:57.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:33:57.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:33:57.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:33:57.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:33:57.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:33:57.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:33:57.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:33:57.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:57.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:57.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:33:57.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:33:57.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:33:57.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:33:57.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:33:57.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:57.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:57.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:33:57.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_lrc_overwrites td/osd-scrub-repair 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:905: TEST_corrupt_and_repair_lrc_overwrites: '[' true = true ']' 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:906: TEST_corrupt_and_repair_lrc_overwrites: corrupt_and_repair_lrc td/osd-scrub-repair true 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:884: corrupt_and_repair_lrc: local dir=td/osd-scrub-repair 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:885: corrupt_and_repair_lrc: local allow_overwrites=true 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:886: corrupt_and_repair_lrc: local poolname=ecpool 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:888: corrupt_and_repair_lrc: run_mon td/osd-scrub-repair a 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:33:57.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:33:57.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:33:57.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:33:57.093 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:33:57.094 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:33:57.094 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:33:57.094 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:33:57.094 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:57.094 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:57.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:57.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:33:57.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:33:57.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:33:57.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:33:57.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:33:57.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:33:57.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:33:57.128 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:33:57.128 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:33:57.128 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:33:57.128 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:33:57.128 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:57.128 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:57.129 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:33:57.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:33:57.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:33:57.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:33:57.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:33:57.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:33:57.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:33:57.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:33:57.185 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:33:57.185 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:33:57.185 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:33:57.186 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:33:57.186 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:57.187 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:57.187 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:33:57.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:33:57.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:33:57.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:889: corrupt_and_repair_lrc: run_mgr td/osd-scrub-repair x 2026-02-22T20:33:57.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:33:57.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:33:57.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:33:57.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:33:57.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:33:57.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:33:57.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:33:57.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:33:57.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:33:57.339 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:33:57.339 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:57.339 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:57.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:57.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:33:57.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:33:57.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: seq 0 9 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 0 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:33:57.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:33:57.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:33:57.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:33:57.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:33:57.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:33:57.362 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:33:57.362 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:33:57.362 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:33:57.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:33:57.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:33:57.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:33:57.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5478f412-d2c2-481a-9dc2-ba4146365f2d 2026-02-22T20:33:57.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5478f412-d2c2-481a-9dc2-ba4146365f2d' 2026-02-22T20:33:57.365 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 5478f412-d2c2-481a-9dc2-ba4146365f2d 2026-02-22T20:33:57.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:33:57.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA1aJtpgp76FRAATs3P1jvrqTphfe/irHl4aw== 2026-02-22T20:33:57.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA1aJtpgp76FRAATs3P1jvrqTphfe/irHl4aw=="}' 2026-02-22T20:33:57.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5478f412-d2c2-481a-9dc2-ba4146365f2d -i td/osd-scrub-repair/0/new.json 2026-02-22T20:33:57.467 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:33:57.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:33:57.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1aJtpgp76FRAATs3P1jvrqTphfe/irHl4aw== --osd-uuid 5478f412-d2c2-481a-9dc2-ba4146365f2d 2026-02-22T20:33:57.492 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:57.483+0000 7f25e0a1aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:57.494 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:57.483+0000 7f25e0a1aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:57.495 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:57.483+0000 7f25e0a1aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:57.495 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:57.483+0000 7f25e0a1aa40 -1 bdev(0x560fff2ecc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:33:57.495 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:57.483+0000 7f25e0a1aa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:33:58.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:33:58.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:33:58.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:33:58.227 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:33:58.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:33:58.432 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:33:58.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:33:58.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:33:58.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:33:58.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:33:58.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:33:58.456 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:58.447+0000 7fdb9c321a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:58.458 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:58.447+0000 7fdb9c321a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:58.459 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:58.447+0000 7fdb9c321a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:33:58.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:33:58.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:33:58.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:33:58.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:33:58.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:33:58.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:58.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:33:58.591 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:33:58.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:58.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:33:58.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:33:58.837 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:58.827+0000 7fdb9c321a40 -1 Falling back to public interface 2026-02-22T20:33:59.442 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:33:59.431+0000 7fdb9c321a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:33:59.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:33:59.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:33:59.752 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:33:59.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:33:59.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:33:59.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:33:59.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:00.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:00.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:00.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:34:00.923 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:34:00.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:00.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:34:01.082 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2764510737,v1:127.0.0.1:6803/2764510737] [v2:127.0.0.1:6804/2764510737,v1:127.0.0.1:6805/2764510737] exists,up 5478f412-d2c2-481a-9dc2-ba4146365f2d 2026-02-22T20:34:01.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:34:01.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:34:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:34:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:34:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 1 2026-02-22T20:34:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:34:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:34:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:34:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:34:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:34:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:34:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:34:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:34:01.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:34:01.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:34:01.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:34:01.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:34:01.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:34:01.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:34:01.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:34:01.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:34:01.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:34:01.086 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:34:01.086 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:34:01.086 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:34:01.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:01.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:01.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:34:01.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:34:01.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:34:01.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:34:01.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:34:01.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:34:01.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:34:01.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:34:01.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:34:01.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:34:01.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:34:01.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:34:01.088 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:34:01.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=09ae89fe-f037-4786-aaa4-dc151c77db4d 2026-02-22T20:34:01.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 09ae89fe-f037-4786-aaa4-dc151c77db4d' 2026-02-22T20:34:01.089 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 09ae89fe-f037-4786-aaa4-dc151c77db4d 2026-02-22T20:34:01.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:34:01.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA5aJtpgHxUBRAANVQ+nDG+x+MlGIOKETEmKQ== 2026-02-22T20:34:01.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA5aJtpgHxUBRAANVQ+nDG+x+MlGIOKETEmKQ=="}' 2026-02-22T20:34:01.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 09ae89fe-f037-4786-aaa4-dc151c77db4d -i td/osd-scrub-repair/1/new.json 2026-02-22T20:34:01.255 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:34:01.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:34:01.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5aJtpgHxUBRAANVQ+nDG+x+MlGIOKETEmKQ== --osd-uuid 09ae89fe-f037-4786-aaa4-dc151c77db4d 2026-02-22T20:34:01.282 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:01.271+0000 7fd9d2520a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:01.284 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:01.275+0000 7fd9d2520a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:01.284 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:01.275+0000 7fd9d2520a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:01.285 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:01.275+0000 7fd9d2520a40 -1 bdev(0x55a8cbe92c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:34:01.285 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:01.275+0000 7fd9d2520a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:34:01.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:34:01.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:34:01.923 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:34:01.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:34:01.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:34:02.120 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:34:02.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:34:02.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:34:02.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:34:02.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:34:02.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:34:02.143 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:02.131+0000 7feb9efd5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:02.145 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:02.135+0000 7feb9efd5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:02.146 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:02.135+0000 7feb9efd5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:02.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:34:02.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:34:02.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:34:02.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:34:02.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:34:02.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:02.284 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:34:02.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:34:02.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:02.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:34:02.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:02.509 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:02.499+0000 7feb9efd5a40 -1 Falling back to public interface 2026-02-22T20:34:03.095 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:03.083+0000 7feb9efd5a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:34:03.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:03.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:03.449 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:34:03.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:34:03.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:03.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:34:03.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:04.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:04.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:04.605 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:34:04.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:34:04.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:04.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:34:04.764 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4205359988,v1:127.0.0.1:6811/4205359988] [v2:127.0.0.1:6812/4205359988,v1:127.0.0.1:6813/4205359988] exists,up 09ae89fe-f037-4786-aaa4-dc151c77db4d 2026-02-22T20:34:04.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:34:04.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 2 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:34:04.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:34:04.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:34:04.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:34:04.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:34:04.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:34:04.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:34:04.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:34:04.768 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:34:04.768 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:34:04.768 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:34:04.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:04.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:04.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:34:04.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:34:04.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:34:04.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:34:04.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:34:04.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:34:04.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:34:04.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:34:04.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:34:04.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:34:04.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:34:04.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:34:04.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:34:04.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0fcd3e98-6f36-469e-b46a-d17c50566bfe 2026-02-22T20:34:04.770 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 0fcd3e98-6f36-469e-b46a-d17c50566bfe 2026-02-22T20:34:04.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 0fcd3e98-6f36-469e-b46a-d17c50566bfe' 2026-02-22T20:34:04.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:34:04.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA8aJtpH5MVLhAA2FIvpZFUPNgnqGoWdIpPeQ== 2026-02-22T20:34:04.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA8aJtpH5MVLhAA2FIvpZFUPNgnqGoWdIpPeQ=="}' 2026-02-22T20:34:04.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0fcd3e98-6f36-469e-b46a-d17c50566bfe -i td/osd-scrub-repair/2/new.json 2026-02-22T20:34:04.947 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:34:04.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:34:04.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8aJtpH5MVLhAA2FIvpZFUPNgnqGoWdIpPeQ== --osd-uuid 0fcd3e98-6f36-469e-b46a-d17c50566bfe 2026-02-22T20:34:04.974 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:04.963+0000 7f3524490a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:04.976 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:04.967+0000 7f3524490a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:04.977 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:04.967+0000 7f3524490a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:04.977 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:04.967+0000 7f3524490a40 -1 bdev(0x563501476c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:34:04.977 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:04.967+0000 7f3524490a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:34:05.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:34:05.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:34:05.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:34:05.636 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:34:05.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:34:05.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:34:05.839 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:34:05.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:34:05.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:34:05.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:34:05.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:34:05.864 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:05.855+0000 7f4e1855fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:05.866 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:05.855+0000 7f4e1855fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:05.867 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:05.855+0000 7f4e1855fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:05.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:34:05.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:34:05.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:34:05.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:34:05.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:34:05.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:05.996 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:34:05.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:34:05.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:05.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:34:06.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:06.249 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:06.239+0000 7f4e1855fa40 -1 Falling back to public interface 2026-02-22T20:34:06.845 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:06.835+0000 7f4e1855fa40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:34:07.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:07.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:07.155 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:34:07.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:34:07.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:07.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:34:07.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:08.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:08.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:08.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:34:08.313 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:34:08.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:08.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:34:08.470 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/406966458,v1:127.0.0.1:6819/406966458] [v2:127.0.0.1:6820/406966458,v1:127.0.0.1:6821/406966458] exists,up 0fcd3e98-6f36-469e-b46a-d17c50566bfe 2026-02-22T20:34:08.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:34:08.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:34:08.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:34:08.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:34:08.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 3 2026-02-22T20:34:08.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:34:08.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:34:08.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-22T20:34:08.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:34:08.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:34:08.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:34:08.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:34:08.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:34:08.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:34:08.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:34:08.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:34:08.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:34:08.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:34:08.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:34:08.475 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:34:08.475 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:34:08.475 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:34:08.475 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:34:08.475 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:34:08.475 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:34:08.475 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:08.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:08.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:34:08.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:34:08.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:34:08.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:34:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:34:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:34:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:34:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:34:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:34:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:34:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:34:08.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:34:08.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:34:08.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e04d4bb4-34bb-453b-844d-af8ddea2016c 2026-02-22T20:34:08.477 INFO:tasks.workunit.client.0.trial150.stdout:add osd3 e04d4bb4-34bb-453b-844d-af8ddea2016c 2026-02-22T20:34:08.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 e04d4bb4-34bb-453b-844d-af8ddea2016c' 2026-02-22T20:34:08.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:34:08.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBAaJtpp/V6HBAA4R7GuCZLXGPjIOUi7twMIA== 2026-02-22T20:34:08.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBAaJtpp/V6HBAA4R7GuCZLXGPjIOUi7twMIA=="}' 2026-02-22T20:34:08.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e04d4bb4-34bb-453b-844d-af8ddea2016c -i td/osd-scrub-repair/3/new.json 2026-02-22T20:34:08.647 INFO:tasks.workunit.client.0.trial150.stdout:3 2026-02-22T20:34:08.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-22T20:34:08.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAaJtpp/V6HBAA4R7GuCZLXGPjIOUi7twMIA== --osd-uuid e04d4bb4-34bb-453b-844d-af8ddea2016c 2026-02-22T20:34:08.674 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:08.663+0000 7fe7dbb4da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:08.675 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:08.667+0000 7fe7dbb4da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:08.677 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:08.667+0000 7fe7dbb4da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:08.677 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:08.667+0000 7fe7dbb4da40 -1 bdev(0x555f88500c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-22T20:34:08.677 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:08.667+0000 7fe7dbb4da40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-22T20:34:09.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-22T20:34:09.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:34:09.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-22T20:34:09.376 INFO:tasks.workunit.client.0.trial150.stdout:adding osd3 key to auth repository 2026-02-22T20:34:09.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:34:09.570 INFO:tasks.workunit.client.0.trial150.stdout:start osd.3 2026-02-22T20:34:09.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-22T20:34:09.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:34:09.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:34:09.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:34:09.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:34:09.595 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:09.583+0000 7f01c1e17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:09.597 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:09.587+0000 7f01c1e17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:09.598 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:09.587+0000 7f01c1e17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:09.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-22T20:34:09.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:34:09.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:34:09.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:34:09.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:34:09.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:09.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:34:09.730 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:34:09.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:09.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:34:09.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:09.969 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:09.959+0000 7f01c1e17a40 -1 Falling back to public interface 2026-02-22T20:34:10.531 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:10.519+0000 7f01c1e17a40 -1 osd.3 0 log_to_monitors true 2026-02-22T20:34:10.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:10.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:10.889 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:34:10.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:34:10.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:10.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:34:11.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:12.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:12.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:12.047 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:34:12.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:34:12.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:12.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:34:12.212 INFO:tasks.workunit.client.0.trial150.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/151230604,v1:127.0.0.1:6827/151230604] [v2:127.0.0.1:6828/151230604,v1:127.0.0.1:6829/151230604] exists,up e04d4bb4-34bb-453b-844d-af8ddea2016c 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 4 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/4 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/4' 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/4/journal' 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:34:12.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:34:12.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:34:12.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:34:12.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:34:12.214 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:34:12.214 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:34:12.214 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:34:12.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:34:12.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/4 2026-02-22T20:34:12.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:34:12.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bb3bfc3e-44a7-4f5a-92f1-bf49d5f843b6 2026-02-22T20:34:12.218 INFO:tasks.workunit.client.0.trial150.stdout:add osd4 bb3bfc3e-44a7-4f5a-92f1-bf49d5f843b6 2026-02-22T20:34:12.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 bb3bfc3e-44a7-4f5a-92f1-bf49d5f843b6' 2026-02-22T20:34:12.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:34:12.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBEaJtpwSYtDRAABxKKL5UGv51eIzs+MbMG5g== 2026-02-22T20:34:12.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBEaJtpwSYtDRAABxKKL5UGv51eIzs+MbMG5g=="}' 2026-02-22T20:34:12.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bb3bfc3e-44a7-4f5a-92f1-bf49d5f843b6 -i td/osd-scrub-repair/4/new.json 2026-02-22T20:34:12.391 INFO:tasks.workunit.client.0.trial150.stdout:4 2026-02-22T20:34:12.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/4/new.json 2026-02-22T20:34:12.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBEaJtpwSYtDRAABxKKL5UGv51eIzs+MbMG5g== --osd-uuid bb3bfc3e-44a7-4f5a-92f1-bf49d5f843b6 2026-02-22T20:34:12.428 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:12.419+0000 7f8f6573fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:12.430 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:12.419+0000 7f8f6573fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:12.431 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:12.419+0000 7f8f6573fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:12.431 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:12.423+0000 7f8f6573fa40 -1 bdev(0x55f65524ac00 td/osd-scrub-repair/4/block) open stat got: (1) Operation not permitted 2026-02-22T20:34:12.431 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:12.423+0000 7f8f6573fa40 -1 bluestore(td/osd-scrub-repair/4) _read_fsid unparsable uuid 2026-02-22T20:34:13.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/4/keyring 2026-02-22T20:34:13.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:34:13.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-22T20:34:13.085 INFO:tasks.workunit.client.0.trial150.stdout:adding osd4 key to auth repository 2026-02-22T20:34:13.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:34:13.282 INFO:tasks.workunit.client.0.trial150.stdout:start osd.4 2026-02-22T20:34:13.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-22T20:34:13.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/4 --osd-journal=td/osd-scrub-repair/4/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:34:13.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:34:13.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:34:13.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:34:13.298 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:13.287+0000 7f1865947a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:13.299 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:13.287+0000 7f1865947a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:13.301 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:13.291+0000 7f1865947a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:13.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-22T20:34:13.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:34:13.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-22T20:34:13.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:34:13.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:34:13.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:13.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:34:13.443 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:34:13.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:13.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-22T20:34:13.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:13.769 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:13.759+0000 7f1865947a40 -1 Falling back to public interface 2026-02-22T20:34:14.417 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:14.407+0000 7f1865947a40 -1 osd.4 0 log_to_monitors true 2026-02-22T20:34:14.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:14.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:14.602 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:34:14.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:34:14.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:14.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-22T20:34:14.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:15.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:15.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:15.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:34:15.759 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:34:15.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:15.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-22T20:34:15.914 INFO:tasks.workunit.client.0.trial150.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1853179020,v1:127.0.0.1:6835/1853179020] [v2:127.0.0.1:6836/1853179020,v1:127.0.0.1:6837/1853179020] exists,up bb3bfc3e-44a7-4f5a-92f1-bf49d5f843b6 2026-02-22T20:34:15.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:34:15.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 5 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:34:15.916 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:34:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/5 2026-02-22T20:34:15.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:34:15.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=24792635-284b-494d-b483-2f02de46b8ba 2026-02-22T20:34:15.919 INFO:tasks.workunit.client.0.trial150.stdout:add osd5 24792635-284b-494d-b483-2f02de46b8ba 2026-02-22T20:34:15.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 24792635-284b-494d-b483-2f02de46b8ba' 2026-02-22T20:34:15.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:34:15.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBHaJtpf9QWNxAAhy8oscY1iKj0PB0NXjeZ3w== 2026-02-22T20:34:15.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBHaJtpf9QWNxAAhy8oscY1iKj0PB0NXjeZ3w=="}' 2026-02-22T20:34:15.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 24792635-284b-494d-b483-2f02de46b8ba -i td/osd-scrub-repair/5/new.json 2026-02-22T20:34:16.092 INFO:tasks.workunit.client.0.trial150.stdout:5 2026-02-22T20:34:16.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/5/new.json 2026-02-22T20:34:16.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHaJtpf9QWNxAAhy8oscY1iKj0PB0NXjeZ3w== --osd-uuid 24792635-284b-494d-b483-2f02de46b8ba 2026-02-22T20:34:16.118 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:16.107+0000 7fe883267a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:16.119 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:16.111+0000 7fe883267a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:16.120 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:16.111+0000 7fe883267a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:16.121 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:16.111+0000 7fe883267a40 -1 bdev(0x55f256ad0c00 td/osd-scrub-repair/5/block) open stat got: (1) Operation not permitted 2026-02-22T20:34:16.121 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:16.111+0000 7fe883267a40 -1 bluestore(td/osd-scrub-repair/5) _read_fsid unparsable uuid 2026-02-22T20:34:16.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/5/keyring 2026-02-22T20:34:16.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:34:16.752 INFO:tasks.workunit.client.0.trial150.stdout:adding osd5 key to auth repository 2026-02-22T20:34:16.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-22T20:34:16.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:34:16.951 INFO:tasks.workunit.client.0.trial150.stdout:start osd.5 2026-02-22T20:34:16.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-22T20:34:16.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:34:16.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:34:16.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:34:16.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:34:16.976 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:16.967+0000 7f48ffccfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:16.978 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:16.967+0000 7f48ffccfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:16.980 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:16.967+0000 7f48ffccfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:17.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-22T20:34:17.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:34:17.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-22T20:34:17.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:34:17.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:34:17.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:17.114 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:34:17.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:34:17.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:17.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:34:17.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:17.349 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:17.339+0000 7f48ffccfa40 -1 Falling back to public interface 2026-02-22T20:34:17.897 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:17.887+0000 7f48ffccfa40 -1 osd.5 0 log_to_monitors true 2026-02-22T20:34:18.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:18.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:18.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:34:18.274 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:34:18.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:18.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:34:18.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:19.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:19.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:19.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:34:19.434 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:34:19.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:19.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:34:19.592 INFO:tasks.workunit.client.0.trial150.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/366147426,v1:127.0.0.1:6843/366147426] [v2:127.0.0.1:6844/366147426,v1:127.0.0.1:6845/366147426] exists,up 24792635-284b-494d-b483-2f02de46b8ba 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 6 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/6 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:34:19.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:34:19.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/6' 2026-02-22T20:34:19.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/6/journal' 2026-02-22T20:34:19.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:34:19.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:34:19.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:34:19.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:34:19.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:34:19.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:34:19.595 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:34:19.595 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:34:19.595 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:34:19.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:34:19.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/6 2026-02-22T20:34:19.597 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:34:19.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=14cd542e-96af-4e4a-8507-83722597988d 2026-02-22T20:34:19.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 14cd542e-96af-4e4a-8507-83722597988d' 2026-02-22T20:34:19.597 INFO:tasks.workunit.client.0.trial150.stdout:add osd6 14cd542e-96af-4e4a-8507-83722597988d 2026-02-22T20:34:19.597 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:34:19.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBLaJtpt4bUIxAAi7ZomvJ0kPftiiP6BLpB/w== 2026-02-22T20:34:19.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBLaJtpt4bUIxAAi7ZomvJ0kPftiiP6BLpB/w=="}' 2026-02-22T20:34:19.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 14cd542e-96af-4e4a-8507-83722597988d -i td/osd-scrub-repair/6/new.json 2026-02-22T20:34:19.763 INFO:tasks.workunit.client.0.trial150.stdout:6 2026-02-22T20:34:19.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/6/new.json 2026-02-22T20:34:19.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBLaJtpt4bUIxAAi7ZomvJ0kPftiiP6BLpB/w== --osd-uuid 14cd542e-96af-4e4a-8507-83722597988d 2026-02-22T20:34:19.801 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:19.791+0000 7f867bec7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:19.803 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:19.791+0000 7f867bec7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:19.804 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:19.795+0000 7f867bec7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:19.804 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:19.795+0000 7f867bec7a40 -1 bdev(0x55ac2d806c00 td/osd-scrub-repair/6/block) open stat got: (1) Operation not permitted 2026-02-22T20:34:19.805 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:19.795+0000 7f867bec7a40 -1 bluestore(td/osd-scrub-repair/6) _read_fsid unparsable uuid 2026-02-22T20:34:20.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/6/keyring 2026-02-22T20:34:20.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:34:20.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-22T20:34:20.461 INFO:tasks.workunit.client.0.trial150.stdout:adding osd6 key to auth repository 2026-02-22T20:34:20.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:34:20.657 INFO:tasks.workunit.client.0.trial150.stdout:start osd.6 2026-02-22T20:34:20.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-22T20:34:20.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/6 --osd-journal=td/osd-scrub-repair/6/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:34:20.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:34:20.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:34:20.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:34:20.673 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:20.663+0000 7f013860aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:20.675 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:20.663+0000 7f013860aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:20.676 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:20.663+0000 7f013860aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:20.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-22T20:34:20.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:34:20.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-22T20:34:20.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:34:20.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:34:20.818 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:34:20.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:20.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:34:20.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:20.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-22T20:34:20.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:21.029 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:21.019+0000 7f013860aa40 -1 Falling back to public interface 2026-02-22T20:34:21.606 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:21.595+0000 7f013860aa40 -1 osd.6 0 log_to_monitors true 2026-02-22T20:34:21.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:21.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:21.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:34:21.982 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:34:21.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:21.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-22T20:34:22.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:23.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:23.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:23.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:34:23.143 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:34:23.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:23.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-22T20:34:23.301 INFO:tasks.workunit.client.0.trial150.stdout:osd.6 up in weight 1 up_from 35 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/850942865,v1:127.0.0.1:6851/850942865] [v2:127.0.0.1:6852/850942865,v1:127.0.0.1:6853/850942865] exists,up 14cd542e-96af-4e4a-8507-83722597988d 2026-02-22T20:34:23.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:34:23.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:34:23.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:34:23.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:34:23.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 7 2026-02-22T20:34:23.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:34:23.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:34:23.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-22T20:34:23.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:34:23.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/7 2026-02-22T20:34:23.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:34:23.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:34:23.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:34:23.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:34:23.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/7' 2026-02-22T20:34:23.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/7/journal' 2026-02-22T20:34:23.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:34:23.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:34:23.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:34:23.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:34:23.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:34:23.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:34:23.306 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:34:23.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:34:23.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:34:23.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:34:23.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/7 2026-02-22T20:34:23.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:34:23.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=bee65a03-fa44-4c22-b178-c22503b9dc7d 2026-02-22T20:34:23.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 bee65a03-fa44-4c22-b178-c22503b9dc7d' 2026-02-22T20:34:23.309 INFO:tasks.workunit.client.0.trial150.stdout:add osd7 bee65a03-fa44-4c22-b178-c22503b9dc7d 2026-02-22T20:34:23.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:34:23.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBPaJtpH0R0EhAAiW1MmMPnBvk7zqUh/Uu8CQ== 2026-02-22T20:34:23.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBPaJtpH0R0EhAAiW1MmMPnBvk7zqUh/Uu8CQ=="}' 2026-02-22T20:34:23.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new bee65a03-fa44-4c22-b178-c22503b9dc7d -i td/osd-scrub-repair/7/new.json 2026-02-22T20:34:23.479 INFO:tasks.workunit.client.0.trial150.stdout:7 2026-02-22T20:34:23.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/7/new.json 2026-02-22T20:34:23.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPaJtpH0R0EhAAiW1MmMPnBvk7zqUh/Uu8CQ== --osd-uuid bee65a03-fa44-4c22-b178-c22503b9dc7d 2026-02-22T20:34:23.506 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:23.495+0000 7f358c493a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:23.507 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:23.495+0000 7f358c493a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:23.508 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:23.499+0000 7f358c493a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:23.509 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:23.499+0000 7f358c493a40 -1 bdev(0x56483896ac00 td/osd-scrub-repair/7/block) open stat got: (1) Operation not permitted 2026-02-22T20:34:23.509 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:23.499+0000 7f358c493a40 -1 bluestore(td/osd-scrub-repair/7) _read_fsid unparsable uuid 2026-02-22T20:34:24.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/7/keyring 2026-02-22T20:34:24.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:34:24.175 INFO:tasks.workunit.client.0.trial150.stdout:adding osd7 key to auth repository 2026-02-22T20:34:24.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-22T20:34:24.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:34:24.380 INFO:tasks.workunit.client.0.trial150.stdout:start osd.7 2026-02-22T20:34:24.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-22T20:34:24.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/7 --osd-journal=td/osd-scrub-repair/7/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:34:24.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:34:24.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:34:24.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:34:24.395 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:24.383+0000 7f436ac13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:24.396 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:24.387+0000 7f436ac13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:24.397 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:24.387+0000 7f436ac13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-22T20:34:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:34:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-22T20:34:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:34:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:34:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:34:24.543 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:34:24.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:24.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-22T20:34:24.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:24.805 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:24.795+0000 7f436ac13a40 -1 Falling back to public interface 2026-02-22T20:34:25.346 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:25.335+0000 7f436ac13a40 -1 osd.7 0 log_to_monitors true 2026-02-22T20:34:25.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:25.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:25.701 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:34:25.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:34:25.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:25.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-22T20:34:25.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:26.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:26.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:26.860 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:34:26.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:34:26.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:26.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-22T20:34:27.014 INFO:tasks.workunit.client.0.trial150.stdout:osd.7 up in weight 1 up_from 40 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3265867769,v1:127.0.0.1:6859/3265867769] [v2:127.0.0.1:6860/3265867769,v1:127.0.0.1:6861/3265867769] exists,up bee65a03-fa44-4c22-b178-c22503b9dc7d 2026-02-22T20:34:27.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:34:27.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:34:27.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:34:27.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:34:27.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 8 2026-02-22T20:34:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:34:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:34:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-22T20:34:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:34:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/8 2026-02-22T20:34:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:34:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:34:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:34:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:34:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/8' 2026-02-22T20:34:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/8/journal' 2026-02-22T20:34:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:34:27.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:34:27.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:34:27.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:34:27.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:34:27.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:34:27.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:34:27.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:34:27.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:34:27.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:34:27.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/8 2026-02-22T20:34:27.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:34:27.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=780b7326-82d8-4b6c-b824-e34c9dbf77cf 2026-02-22T20:34:27.022 INFO:tasks.workunit.client.0.trial150.stdout:add osd8 780b7326-82d8-4b6c-b824-e34c9dbf77cf 2026-02-22T20:34:27.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 780b7326-82d8-4b6c-b824-e34c9dbf77cf' 2026-02-22T20:34:27.023 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:34:27.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBTaJtpvhhQARAAfK34Fon/juIie7AjAWLZwQ== 2026-02-22T20:34:27.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBTaJtpvhhQARAAfK34Fon/juIie7AjAWLZwQ=="}' 2026-02-22T20:34:27.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 780b7326-82d8-4b6c-b824-e34c9dbf77cf -i td/osd-scrub-repair/8/new.json 2026-02-22T20:34:27.194 INFO:tasks.workunit.client.0.trial150.stdout:8 2026-02-22T20:34:27.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/8/new.json 2026-02-22T20:34:27.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTaJtpvhhQARAAfK34Fon/juIie7AjAWLZwQ== --osd-uuid 780b7326-82d8-4b6c-b824-e34c9dbf77cf 2026-02-22T20:34:27.221 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:27.211+0000 7f6c27aa6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:27.223 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:27.211+0000 7f6c27aa6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:27.223 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:27.211+0000 7f6c27aa6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:27.224 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:27.215+0000 7f6c27aa6a40 -1 bdev(0x55e3b79e8c00 td/osd-scrub-repair/8/block) open stat got: (1) Operation not permitted 2026-02-22T20:34:27.224 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:27.215+0000 7f6c27aa6a40 -1 bluestore(td/osd-scrub-repair/8) _read_fsid unparsable uuid 2026-02-22T20:34:27.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/8/keyring 2026-02-22T20:34:27.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:34:27.908 INFO:tasks.workunit.client.0.trial150.stdout:adding osd8 key to auth repository 2026-02-22T20:34:27.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-22T20:34:27.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:34:28.107 INFO:tasks.workunit.client.0.trial150.stdout:start osd.8 2026-02-22T20:34:28.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-22T20:34:28.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/8 --osd-journal=td/osd-scrub-repair/8/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:34:28.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:34:28.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:34:28.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:34:28.132 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:28.119+0000 7f3e9ea25a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:28.133 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:28.123+0000 7f3e9ea25a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:28.134 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:28.123+0000 7f3e9ea25a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:28.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-22T20:34:28.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:34:28.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-22T20:34:28.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:34:28.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:34:28.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:28.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:34:28.271 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:34:28.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:28.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-22T20:34:28.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:28.522 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:28.511+0000 7f3e9ea25a40 -1 Falling back to public interface 2026-02-22T20:34:29.066 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:29.055+0000 7f3e9ea25a40 -1 osd.8 0 log_to_monitors true 2026-02-22T20:34:29.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:29.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:29.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:34:29.431 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:34:29.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:29.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-22T20:34:29.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:30.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:30.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:34:30.595 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:34:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-22T20:34:30.750 INFO:tasks.workunit.client.0.trial150.stdout:osd.8 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/2584130223,v1:127.0.0.1:6867/2584130223] [v2:127.0.0.1:6868/2584130223,v1:127.0.0.1:6869/2584130223] exists,up 780b7326-82d8-4b6c-b824-e34c9dbf77cf 2026-02-22T20:34:30.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:34:30.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:34:30.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:34:30.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:890: corrupt_and_repair_lrc: for id in $(seq 0 9) 2026-02-22T20:34:30.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:891: corrupt_and_repair_lrc: run_osd td/osd-scrub-repair 9 2026-02-22T20:34:30.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:34:30.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:34:30.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2026-02-22T20:34:30.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:34:30.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:34:30.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:34:30.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:34:30.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:34:30.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:34:30.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-22T20:34:30.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-22T20:34:30.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:34:30.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:34:30.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:34:30.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:34:30.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:34:30.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:34:30.757 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:34:30.757 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:34:30.757 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:34:30.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:30.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:30.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:34:30.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:34:30.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:34:30.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:34:30.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:34:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:34:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:34:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:34:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:34:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:34:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:34:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/9 2026-02-22T20:34:30.758 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:34:30.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f5e55fab-b8f1-4b4c-9e7c-53aa88ccb78f 2026-02-22T20:34:30.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 f5e55fab-b8f1-4b4c-9e7c-53aa88ccb78f' 2026-02-22T20:34:30.760 INFO:tasks.workunit.client.0.trial150.stdout:add osd9 f5e55fab-b8f1-4b4c-9e7c-53aa88ccb78f 2026-02-22T20:34:30.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:34:30.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBWaJtpygo2LRAACsRP6tGZnamD7ppSoGhq3A== 2026-02-22T20:34:30.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBWaJtpygo2LRAACsRP6tGZnamD7ppSoGhq3A=="}' 2026-02-22T20:34:30.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f5e55fab-b8f1-4b4c-9e7c-53aa88ccb78f -i td/osd-scrub-repair/9/new.json 2026-02-22T20:34:30.929 INFO:tasks.workunit.client.0.trial150.stdout:9 2026-02-22T20:34:30.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/9/new.json 2026-02-22T20:34:30.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWaJtpygo2LRAACsRP6tGZnamD7ppSoGhq3A== --osd-uuid f5e55fab-b8f1-4b4c-9e7c-53aa88ccb78f 2026-02-22T20:34:30.955 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:30.943+0000 7f997d3e6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:30.956 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:30.947+0000 7f997d3e6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:30.957 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:30.947+0000 7f997d3e6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:30.958 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:30.947+0000 7f997d3e6a40 -1 bdev(0x5639cafc6c00 td/osd-scrub-repair/9/block) open stat got: (1) Operation not permitted 2026-02-22T20:34:30.958 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:30.947+0000 7f997d3e6a40 -1 bluestore(td/osd-scrub-repair/9) _read_fsid unparsable uuid 2026-02-22T20:34:31.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/9/keyring 2026-02-22T20:34:31.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:34:31.631 INFO:tasks.workunit.client.0.trial150.stdout:adding osd9 key to auth repository 2026-02-22T20:34:31.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2026-02-22T20:34:31.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:34:31.831 INFO:tasks.workunit.client.0.trial150.stdout:start osd.9 2026-02-22T20:34:31.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2026-02-22T20:34:31.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:34:31.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:34:31.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:34:31.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:34:31.856 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:31.847+0000 7fc5efc2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:31.858 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:31.847+0000 7fc5efc2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:31.859 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:31.847+0000 7fc5efc2aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:31.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2026-02-22T20:34:31.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:34:31.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-22T20:34:31.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:34:31.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:34:31.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:31.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:34:31.991 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:34:31.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:31.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:34:32.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:32.266 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:32.255+0000 7fc5efc2aa40 -1 Falling back to public interface 2026-02-22T20:34:32.838 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:32.827+0000 7fc5efc2aa40 -1 osd.9 0 log_to_monitors true 2026-02-22T20:34:33.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:33.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:33.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:34:33.153 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:34:33.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:33.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:34:33.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:34.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:34.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:34.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:34:34.313 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:34:34.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:34.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:34:34.466 INFO:tasks.workunit.client.0.trial150.stdout:osd.9 up in weight 1 up_from 49 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2651282590,v1:127.0.0.1:6875/2651282590] [v2:127.0.0.1:6876/2651282590,v1:127.0.0.1:6877/2651282590] exists,up f5e55fab-b8f1-4b4c-9e7c-53aa88ccb78f 2026-02-22T20:34:34.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:34:34.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:34:34.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:34:34.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:893: corrupt_and_repair_lrc: create_rbd_pool 2026-02-22T20:34:34.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:34:34.611 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:34:34.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:34:34.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:34:34.829 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:34:34.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:34:35.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:34:36.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:894: corrupt_and_repair_lrc: wait_for_clean 2026-02-22T20:34:36.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:34:36.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:34:36.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:34:36.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:34:36.122 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:34:36.122 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:34:36.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:34:36.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:34:36.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:34:36.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:34:36.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:34:36.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:34:36.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:34:36.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:34:36.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:34:36.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:34:36.396 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:34:36.397 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:34:36.397 INFO:tasks.workunit.client.0.trial150.stderr:3 2026-02-22T20:34:36.397 INFO:tasks.workunit.client.0.trial150.stderr:4 2026-02-22T20:34:36.397 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:34:36.397 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:34:36.397 INFO:tasks.workunit.client.0.trial150.stderr:7 2026-02-22T20:34:36.398 INFO:tasks.workunit.client.0.trial150.stderr:8 2026-02-22T20:34:36.398 INFO:tasks.workunit.client.0.trial150.stderr:9' 2026-02-22T20:34:36.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:34:36.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:36.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:34:36.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-22T20:34:36.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-22T20:34:36.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-22T20:34:36.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:36.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:34:36.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672968 2026-02-22T20:34:36.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672968 2026-02-22T20:34:36.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2026-02-22T20:34:36.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:36.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:34:36.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509447 2026-02-22T20:34:36.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509447 2026-02-22T20:34:36.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447' 2026-02-22T20:34:36.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:36.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:34:36.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345926 2026-02-22T20:34:36.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345926 2026-02-22T20:34:36.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-85899345926' 2026-02-22T20:34:36.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:36.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:34:36.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182406 2026-02-22T20:34:36.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182406 2026-02-22T20:34:36.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-85899345926 4-107374182406' 2026-02-22T20:34:36.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:36.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:34:36.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018885 2026-02-22T20:34:36.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018885 2026-02-22T20:34:36.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-85899345926 4-107374182406 5-128849018885' 2026-02-22T20:34:36.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:36.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:34:36.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855364 2026-02-22T20:34:36.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855364 2026-02-22T20:34:36.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-85899345926 4-107374182406 5-128849018885 6-150323855364' 2026-02-22T20:34:36.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:36.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:34:36.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691844 2026-02-22T20:34:36.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691844 2026-02-22T20:34:36.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-85899345926 4-107374182406 5-128849018885 6-150323855364 7-171798691844' 2026-02-22T20:34:36.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:36.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:34:37.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528323 2026-02-22T20:34:37.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528323 2026-02-22T20:34:37.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-85899345926 4-107374182406 5-128849018885 6-150323855364 7-171798691844 8-193273528323' 2026-02-22T20:34:37.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:37.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:34:37.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397507 2026-02-22T20:34:37.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397507 2026-02-22T20:34:37.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509447 3-85899345926 4-107374182406 5-128849018885 6-150323855364 7-171798691844 8-193273528323 9-210453397507' 2026-02-22T20:34:37.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:37.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-22T20:34:37.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:37.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:34:37.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-22T20:34:37.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:37.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-22T20:34:37.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-22T20:34:37.123 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836489 2026-02-22T20:34:37.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:34:37.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836489 2026-02-22T20:34:37.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:34:38.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:34:38.292 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:34:38.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-22T20:34:38.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:38.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672968 2026-02-22T20:34:38.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:38.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:34:38.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672968 2026-02-22T20:34:38.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:38.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672968 2026-02-22T20:34:38.456 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672968 2026-02-22T20:34:38.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2026-02-22T20:34:38.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:34:38.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672968 2026-02-22T20:34:38.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:38.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509447 2026-02-22T20:34:38.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:38.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:34:38.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509447 2026-02-22T20:34:38.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:38.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509447 2026-02-22T20:34:38.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2026-02-22T20:34:38.626 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509447 2026-02-22T20:34:38.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:34:38.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509447 -lt 64424509447 2026-02-22T20:34:38.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:38.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-85899345926 2026-02-22T20:34:38.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:38.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:34:38.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-85899345926 2026-02-22T20:34:38.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:38.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345926 2026-02-22T20:34:38.801 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.3 seq 85899345926 2026-02-22T20:34:38.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2026-02-22T20:34:38.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:34:38.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345927 -lt 85899345926 2026-02-22T20:34:38.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:38.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182406 2026-02-22T20:34:38.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:38.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:34:38.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182406 2026-02-22T20:34:38.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:38.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182406 2026-02-22T20:34:38.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182406' 2026-02-22T20:34:38.973 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.4 seq 107374182406 2026-02-22T20:34:38.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:34:39.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182406 -lt 107374182406 2026-02-22T20:34:39.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:39.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-128849018885 2026-02-22T20:34:39.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:39.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:34:39.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-128849018885 2026-02-22T20:34:39.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:39.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018885 2026-02-22T20:34:39.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 128849018885' 2026-02-22T20:34:39.138 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.5 seq 128849018885 2026-02-22T20:34:39.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:34:39.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018885 -lt 128849018885 2026-02-22T20:34:39.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:39.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855364 2026-02-22T20:34:39.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:39.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:34:39.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855364 2026-02-22T20:34:39.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:39.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855364 2026-02-22T20:34:39.302 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.6 seq 150323855364 2026-02-22T20:34:39.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855364' 2026-02-22T20:34:39.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:34:39.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855365 -lt 150323855364 2026-02-22T20:34:39.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:39.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691844 2026-02-22T20:34:39.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:39.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:34:39.472 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691844 2026-02-22T20:34:39.472 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:39.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691844 2026-02-22T20:34:39.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691844' 2026-02-22T20:34:39.474 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.7 seq 171798691844 2026-02-22T20:34:39.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:34:39.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691844 -lt 171798691844 2026-02-22T20:34:39.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:39.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528323 2026-02-22T20:34:39.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:39.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:34:39.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528323 2026-02-22T20:34:39.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:39.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528323 2026-02-22T20:34:39.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528323' 2026-02-22T20:34:39.645 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.8 seq 193273528323 2026-02-22T20:34:39.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:34:39.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528323 -lt 193273528323 2026-02-22T20:34:39.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:39.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-210453397507 2026-02-22T20:34:39.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:39.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:34:39.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-210453397507 2026-02-22T20:34:39.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:39.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=210453397507 2026-02-22T20:34:39.824 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.9 seq 210453397507 2026-02-22T20:34:39.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 210453397507' 2026-02-22T20:34:39.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:34:39.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 210453397508 -lt 210453397507 2026-02-22T20:34:39.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:34:39.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:34:39.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:34:40.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:34:40.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:34:40.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:34:40.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:34:40.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:34:40.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:34:40.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:34:40.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:34:40.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:34:40.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:34:40.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:34:40.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:34:40.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:34:40.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:34:40.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:34:40.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:896: corrupt_and_repair_lrc: create_ec_pool ecpool true k=4 m=2 l=3 plugin=lrc 2026-02-22T20:34:40.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-22T20:34:40.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:34:40.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-22T20:34:40.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:34:40.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=4 m=2 l=3 plugin=lrc 2026-02-22T20:34:40.752 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:34:40.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-22T20:34:40.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-22T20:34:41.059 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' created 2026-02-22T20:34:41.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:34:42.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-22T20:34:42.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-22T20:34:42.230 INFO:tasks.workunit.client.0.trial150.stderr:set pool 2 allow_ec_overwrites to true 2026-02-22T20:34:42.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:34:42.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:34:42.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:34:42.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:34:42.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:34:42.242 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:34:42.242 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:34:42.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:34:42.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:34:42.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:34:42.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:34:42.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:34:42.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:34:42.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:34:42.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:34:42.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr:3 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr:4 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr:7 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr:8 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr:9' 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:42.507 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:34:42.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-22T20:34:42.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-22T20:34:42.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T20:34:42.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:42.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:34:42.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672970 2026-02-22T20:34:42.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672970 2026-02-22T20:34:42.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2026-02-22T20:34:42.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:42.652 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:34:42.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509449 2026-02-22T20:34:42.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509449 2026-02-22T20:34:42.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449' 2026-02-22T20:34:42.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:42.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:34:42.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345929 2026-02-22T20:34:42.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345929 2026-02-22T20:34:42.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345929' 2026-02-22T20:34:42.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:42.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:34:42.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182408 2026-02-22T20:34:42.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182408 2026-02-22T20:34:42.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345929 4-107374182408' 2026-02-22T20:34:42.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:42.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:34:42.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018887 2026-02-22T20:34:42.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018887 2026-02-22T20:34:42.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345929 4-107374182408 5-128849018887' 2026-02-22T20:34:42.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:42.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:34:43.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855367 2026-02-22T20:34:43.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855367 2026-02-22T20:34:43.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345929 4-107374182408 5-128849018887 6-150323855367' 2026-02-22T20:34:43.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:43.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:34:43.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691846 2026-02-22T20:34:43.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691846 2026-02-22T20:34:43.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345929 4-107374182408 5-128849018887 6-150323855367 7-171798691846' 2026-02-22T20:34:43.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:43.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:34:43.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528325 2026-02-22T20:34:43.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528325 2026-02-22T20:34:43.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345929 4-107374182408 5-128849018887 6-150323855367 7-171798691846 8-193273528325' 2026-02-22T20:34:43.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:43.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:34:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397509 2026-02-22T20:34:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397509 2026-02-22T20:34:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345929 4-107374182408 5-128849018887 6-150323855367 7-171798691846 8-193273528325 9-210453397509' 2026-02-22T20:34:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:43.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-22T20:34:43.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:43.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:34:43.229 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-22T20:34:43.230 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:43.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-22T20:34:43.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T20:34:43.232 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836491 2026-02-22T20:34:43.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:34:43.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T20:34:43.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:34:44.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:34:44.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:34:44.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-22T20:34:44.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:44.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672970 2026-02-22T20:34:44.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:44.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:34:44.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672970 2026-02-22T20:34:44.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:44.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672970 2026-02-22T20:34:44.561 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672970 2026-02-22T20:34:44.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2026-02-22T20:34:44.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:34:44.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672970 -lt 42949672970 2026-02-22T20:34:44.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:44.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509449 2026-02-22T20:34:44.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:44.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:34:44.721 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509449 2026-02-22T20:34:44.721 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:44.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509449 2026-02-22T20:34:44.723 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509449 2026-02-22T20:34:44.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2026-02-22T20:34:44.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:34:44.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509450 -lt 64424509449 2026-02-22T20:34:44.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:44.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-85899345929 2026-02-22T20:34:44.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:44.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:34:44.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-85899345929 2026-02-22T20:34:44.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:44.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345929 2026-02-22T20:34:44.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2026-02-22T20:34:44.885 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.3 seq 85899345929 2026-02-22T20:34:44.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:34:45.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345929 -lt 85899345929 2026-02-22T20:34:45.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:45.057 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182408 2026-02-22T20:34:45.057 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:45.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:34:45.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182408 2026-02-22T20:34:45.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182408 2026-02-22T20:34:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182408' 2026-02-22T20:34:45.061 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.4 seq 107374182408 2026-02-22T20:34:45.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:34:45.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182408 -lt 107374182408 2026-02-22T20:34:45.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:45.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-128849018887 2026-02-22T20:34:45.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:45.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:34:45.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-128849018887 2026-02-22T20:34:45.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:45.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018887 2026-02-22T20:34:45.221 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.5 seq 128849018887 2026-02-22T20:34:45.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2026-02-22T20:34:45.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:34:45.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018887 -lt 128849018887 2026-02-22T20:34:45.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:45.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855367 2026-02-22T20:34:45.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:45.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:34:45.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855367 2026-02-22T20:34:45.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:45.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855367 2026-02-22T20:34:45.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855367' 2026-02-22T20:34:45.386 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.6 seq 150323855367 2026-02-22T20:34:45.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:34:45.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855367 -lt 150323855367 2026-02-22T20:34:45.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:45.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691846 2026-02-22T20:34:45.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:45.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:34:45.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691846 2026-02-22T20:34:45.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:45.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691846 2026-02-22T20:34:45.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691846' 2026-02-22T20:34:45.559 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.7 seq 171798691846 2026-02-22T20:34:45.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:34:45.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691846 -lt 171798691846 2026-02-22T20:34:45.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:45.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528325 2026-02-22T20:34:45.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:45.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:34:45.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528325 2026-02-22T20:34:45.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:45.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528325 2026-02-22T20:34:45.733 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.8 seq 193273528325 2026-02-22T20:34:45.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528325' 2026-02-22T20:34:45.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:34:45.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528326 -lt 193273528325 2026-02-22T20:34:45.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:45.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-210453397509 2026-02-22T20:34:45.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:45.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:34:45.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-210453397509 2026-02-22T20:34:45.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:45.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=210453397509 2026-02-22T20:34:45.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 210453397509' 2026-02-22T20:34:45.895 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.9 seq 210453397509 2026-02-22T20:34:45.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:34:46.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 210453397510 -lt 210453397509 2026-02-22T20:34:46.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:34:46.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:34:46.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:34:46.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:34:46.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:34:46.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:34:46.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:34:46.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:34:46.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:34:46.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:34:46.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:34:46.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:34:46.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:34:46.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:34:46.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:34:46.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:34:46.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:34:46.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:34:46.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:34:46.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:897: corrupt_and_repair_lrc: corrupt_and_repair_erasure_coded td/osd-scrub-repair ecpool 2026-02-22T20:34:46.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:308: corrupt_and_repair_erasure_coded: local dir=td/osd-scrub-repair 2026-02-22T20:34:46.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:309: corrupt_and_repair_erasure_coded: local poolname=ecpool 2026-02-22T20:34:46.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:311: corrupt_and_repair_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-22T20:34:46.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:34:46.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:34:46.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-22T20:34:46.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:34:46.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:34:46.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:34:46.809 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:34:46.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:34:47.013 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:34:47.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:34:47.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:34:47.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:34:47.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: get_primary ecpool SOMETHING 2026-02-22T20:34:47.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-22T20:34:47.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:34:47.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:34:47.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:34:47.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:313: corrupt_and_repair_erasure_coded: local primary=3 2026-02-22T20:34:47.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: get_osds ecpool SOMETHING 2026-02-22T20:34:47.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: sed -e s/3// 2026-02-22T20:34:47.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-22T20:34:47.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-22T20:34:47.213 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:34:47.213 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:34:47.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=3 2026-02-22T20:34:47.375 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:34:47.375 INFO:tasks.workunit.client.0.trial150.stderr:9 2026-02-22T20:34:47.375 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:34:47.375 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:34:47.375 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:34:47.376 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:34:47.376 INFO:tasks.workunit.client.0.trial150.stderr:7' 2026-02-22T20:34:47.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 3 5 9 0 6 2 1 7 2026-02-22T20:34:47.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: osds=('5' '9' '0' '6' '2' '1' '7') 2026-02-22T20:34:47.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:314: corrupt_and_repair_erasure_coded: local -a osds 2026-02-22T20:34:47.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:315: corrupt_and_repair_erasure_coded: local not_primary_first=5 2026-02-22T20:34:47.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:316: corrupt_and_repair_erasure_coded: local not_primary_second=9 2026-02-22T20:34:47.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:319: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 3 2026-02-22T20:34:47.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-22T20:34:47.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-22T20:34:47.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=3 2026-02-22T20:34:47.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:34:47.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:34:47.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:34:47.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:34:47.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:34:47.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:34:47.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:34:47.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:34:47.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:34:47.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:34:47.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:34:47.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:34:47.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:34:47.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:34:47.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:34:47.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:34:47.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:34:47.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:34:47.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:34:47.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:34:47.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:34:47.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:34:47.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:34:47.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-22T20:34:47.901 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#2:eb822e21:::SOMETHING:head# 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:34:48.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:34:48.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:34:48.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:34:48.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:34:48.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:34:48.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:34:48.026 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:34:48.026 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:34:48.026 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:34:48.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:34:48.027 INFO:tasks.workunit.client.0.trial150.stderr:start osd.3 2026-02-22T20:34:48.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:34:48.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:34:48.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:34:48.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:34:48.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:34:48.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:34:48.047 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:48.034+0000 7f27022a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:48.048 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:48.038+0000 7f27022a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:48.049 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:48.038+0000 7f27022a6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:34:48.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:34:48.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:34:48.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:34:48.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:34:48.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:34:48.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:48.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:34:48.186 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:34:48.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:48.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:34:48.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:34:48.450 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:48.438+0000 7f27022a6a40 -1 Falling back to public interface 2026-02-22T20:34:49.118 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:34:49.106+0000 7f27022a6a40 -1 osd.3 66 log_to_monitors true 2026-02-22T20:34:49.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:34:49.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:34:49.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:34:49.349 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:34:49.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:34:49.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:34:49.505 INFO:tasks.workunit.client.0.trial150.stderr:osd.3 up in weight 1 up_from 70 up_thru 70 down_at 67 last_clean_interval [20,66) [v2:127.0.0.1:6826/3646459495,v1:127.0.0.1:6827/3646459495] [v2:127.0.0.1:6828/3646459495,v1:127.0.0.1:6829/3646459495] exists,up e04d4bb4-34bb-453b-844d-af8ddea2016c 2026-02-22T20:34:49.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:34:49.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:34:49.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:34:49.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:34:49.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:34:49.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:34:49.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:34:49.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:34:49.507 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:34:49.507 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:34:49.507 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:34:49.507 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:34:49.507 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:34:49.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:34:49.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:34:49.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:34:49.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:34:49.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:34:49.613 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:34:49.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:34:49.768 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:34:49.768 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:34:49.768 INFO:tasks.workunit.client.0.trial150.stderr:3 2026-02-22T20:34:49.768 INFO:tasks.workunit.client.0.trial150.stderr:4 2026-02-22T20:34:49.769 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:34:49.769 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:34:49.769 INFO:tasks.workunit.client.0.trial150.stderr:7 2026-02-22T20:34:49.769 INFO:tasks.workunit.client.0.trial150.stderr:8 2026-02-22T20:34:49.769 INFO:tasks.workunit.client.0.trial150.stderr:9' 2026-02-22T20:34:49.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:34:49.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:49.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:34:49.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-22T20:34:49.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-22T20:34:49.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-22T20:34:49.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:49.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:34:49.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672973 2026-02-22T20:34:49.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672973 2026-02-22T20:34:49.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973' 2026-02-22T20:34:49.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:49.916 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:34:49.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509452 2026-02-22T20:34:49.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509452 2026-02-22T20:34:49.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452' 2026-02-22T20:34:49.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:49.988 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:34:50.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710722 2026-02-22T20:34:50.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710722 2026-02-22T20:34:50.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-300647710722' 2026-02-22T20:34:50.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:50.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:34:50.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182410 2026-02-22T20:34:50.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182410 2026-02-22T20:34:50.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-300647710722 4-107374182410' 2026-02-22T20:34:50.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:50.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:34:50.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018890 2026-02-22T20:34:50.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018890 2026-02-22T20:34:50.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-300647710722 4-107374182410 5-128849018890' 2026-02-22T20:34:50.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:50.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:34:50.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855369 2026-02-22T20:34:50.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855369 2026-02-22T20:34:50.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-300647710722 4-107374182410 5-128849018890 6-150323855369' 2026-02-22T20:34:50.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:50.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:34:50.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691848 2026-02-22T20:34:50.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691848 2026-02-22T20:34:50.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-300647710722 4-107374182410 5-128849018890 6-150323855369 7-171798691848' 2026-02-22T20:34:50.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:50.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:34:50.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528328 2026-02-22T20:34:50.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528328 2026-02-22T20:34:50.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-300647710722 4-107374182410 5-128849018890 6-150323855369 7-171798691848 8-193273528328' 2026-02-22T20:34:50.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:34:50.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:34:50.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=210453397512 2026-02-22T20:34:50.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 210453397512 2026-02-22T20:34:50.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509452 3-300647710722 4-107374182410 5-128849018890 6-150323855369 7-171798691848 8-193273528328 9-210453397512' 2026-02-22T20:34:50.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:50.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-22T20:34:50.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:50.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:34:50.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-22T20:34:50.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:50.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-22T20:34:50.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-22T20:34:50.492 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836493 2026-02-22T20:34:50.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:34:50.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-22T20:34:50.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:50.652 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672973 2026-02-22T20:34:50.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:50.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:34:50.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672973 2026-02-22T20:34:50.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:50.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672973 2026-02-22T20:34:50.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2026-02-22T20:34:50.656 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672973 2026-02-22T20:34:50.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:34:50.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672972 -lt 42949672973 2026-02-22T20:34:50.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:34:51.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:34:51.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:34:51.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672973 -lt 42949672973 2026-02-22T20:34:51.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:51.988 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509452 2026-02-22T20:34:51.989 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:51.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:34:51.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509452 2026-02-22T20:34:51.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:51.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509452 2026-02-22T20:34:51.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2026-02-22T20:34:51.992 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509452 2026-02-22T20:34:51.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:34:52.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509452 -lt 64424509452 2026-02-22T20:34:52.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:52.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-300647710722 2026-02-22T20:34:52.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:52.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:34:52.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-300647710722 2026-02-22T20:34:52.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:52.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710722 2026-02-22T20:34:52.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 300647710722' 2026-02-22T20:34:52.158 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 300647710722 2026-02-22T20:34:52.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:34:52.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710722 -lt 300647710722 2026-02-22T20:34:52.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:52.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182410 2026-02-22T20:34:52.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:52.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:34:52.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182410 2026-02-22T20:34:52.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:52.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182410 2026-02-22T20:34:52.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2026-02-22T20:34:52.334 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.4 seq 107374182410 2026-02-22T20:34:52.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:34:52.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182411 -lt 107374182410 2026-02-22T20:34:52.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:52.507 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-128849018890 2026-02-22T20:34:52.507 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:52.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:34:52.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-128849018890 2026-02-22T20:34:52.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:52.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018890 2026-02-22T20:34:52.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 128849018890' 2026-02-22T20:34:52.511 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.5 seq 128849018890 2026-02-22T20:34:52.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:34:52.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018890 -lt 128849018890 2026-02-22T20:34:52.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:52.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855369 2026-02-22T20:34:52.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:52.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:34:52.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855369 2026-02-22T20:34:52.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:52.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855369 2026-02-22T20:34:52.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855369' 2026-02-22T20:34:52.677 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.6 seq 150323855369 2026-02-22T20:34:52.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:34:52.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855369 -lt 150323855369 2026-02-22T20:34:52.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:52.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691848 2026-02-22T20:34:52.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:52.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:34:52.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691848 2026-02-22T20:34:52.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:52.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691848 2026-02-22T20:34:52.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691848' 2026-02-22T20:34:52.841 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.7 seq 171798691848 2026-02-22T20:34:52.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:34:53.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691849 -lt 171798691848 2026-02-22T20:34:53.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:53.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528328 2026-02-22T20:34:53.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:53.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:34:53.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528328 2026-02-22T20:34:53.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:53.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528328 2026-02-22T20:34:53.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528328' 2026-02-22T20:34:53.005 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.8 seq 193273528328 2026-02-22T20:34:53.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:34:53.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528328 -lt 193273528328 2026-02-22T20:34:53.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:34:53.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-210453397512 2026-02-22T20:34:53.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:34:53.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:34:53.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-210453397512 2026-02-22T20:34:53.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:34:53.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=210453397512 2026-02-22T20:34:53.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 210453397512' 2026-02-22T20:34:53.174 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.9 seq 210453397512 2026-02-22T20:34:53.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:34:53.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 210453397512 -lt 210453397512 2026-02-22T20:34:53.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:34:53.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:34:53.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:34:53.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:34:53.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:34:53.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:34:53.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:34:53.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:34:53.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:34:53.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:34:53.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:34:53.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:34:53.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:34:53.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:34:53.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:34:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:34:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:34:53.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:34:53.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-22T20:34:53.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:34:53.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:34:53.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:34:53.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:34:54.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-22T20:34:54.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-22T20:34:54.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:34:54.024 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:34:54.024 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:34:54.024 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:34:54.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:34:54.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:34:54.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:34:41.048726+0000 2026-02-22T20:34:54.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:34:54.330 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:34:54.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:34:41.048726+0000 2026-02-22T20:34:54.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:34:54.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:34:41.048726+0000 2026-02-22T20:34:54.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:34:54.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:34:54.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:34:54.341 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:34:54.341 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:34:54.341 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:34:54.342 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:34:54.342 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:34:54.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:34:41.048726+0000 '>' 2026-02-22T20:34:41.048726+0000 2026-02-22T20:34:54.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:34:55.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:34:55.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:34:55.503 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:34:55.503 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:34:55.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:34:55.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:34:55.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:34:55.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:34:41.048726+0000 '>' 2026-02-22T20:34:41.048726+0000 2026-02-22T20:34:55.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:34:56.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:34:56.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:34:56.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:34:56.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:34:56.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:34:56.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:34:56.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:34:56.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:34:41.048726+0000 '>' 2026-02-22T20:34:41.048726+0000 2026-02-22T20:34:56.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:34:57.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:34:57.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:34:57.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:34:57.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:34:57.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:34:57.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:34:57.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:34:57.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:34:41.048726+0000 '>' 2026-02-22T20:34:41.048726+0000 2026-02-22T20:34:57.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:34:58.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:34:58.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:34:58.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:34:58.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:34:58.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:34:58.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:34:58.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:34:59.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:34:41.048726+0000 '>' 2026-02-22T20:34:41.048726+0000 2026-02-22T20:34:59.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:00.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:00.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:00.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:00.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:00.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:00.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:00.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:00.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:34:55.044413+0000 '>' 2026-02-22T20:34:41.048726+0000 2026-02-22T20:35:00.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:35:00.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:35:00.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:35:00.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:35:00.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-22T20:35:00.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:35:00.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:35:00.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:35:00.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:35:00.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-22T20:35:00.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:35:00.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-22T20:35:00.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:35:00.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:35:00.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:35:00.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:35:00.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:35:00.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:35:00.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:35:00.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:35:00.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:35:00.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-22T20:35:00.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:35:00.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:35:00.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-22T20:35:00.699 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:35:00.699 INFO:tasks.workunit.client.0.trial150.stdout:hinfo_key 2026-02-22T20:35:00.700 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:35:00.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:35:00.785 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:35:00.786 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:35:00.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:00.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:00.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:35:00.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:35:00.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:35:00.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:35:00.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:35:00.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:35:00.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:35:00.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:35:00.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:35:00.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-22T20:35:00.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-22T20:35:00.787 INFO:tasks.workunit.client.0.trial150.stderr:start osd.9 2026-02-22T20:35:00.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:35:00.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-22T20:35:00.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-22T20:35:00.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:35:00.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:35:00.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:35:00.805 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:00.794+0000 7fcc2c171a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:00.807 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:00.794+0000 7fcc2c171a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:00.808 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:00.794+0000 7fcc2c171a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:00.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-22T20:35:00.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:35:00.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-22T20:35:00.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:35:00.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:35:00.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:00.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:35:00.949 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:35:00.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:00.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:35:01.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:35:01.227 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:01.214+0000 7fcc2c171a40 -1 Falling back to public interface 2026-02-22T20:35:01.912 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:01.902+0000 7fcc2c171a40 -1 osd.9 72 log_to_monitors true 2026-02-22T20:35:02.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:35:02.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:02.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:35:02.109 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:35:02.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:02.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:35:02.268 INFO:tasks.workunit.client.0.trial150.stderr:osd.9 up in weight 1 up_from 75 up_thru 0 down_at 72 last_clean_interval [49,72) [v2:127.0.0.1:6874/1394729797,v1:127.0.0.1:6875/1394729797] [v2:127.0.0.1:6876/1394729797,v1:127.0.0.1:6877/1394729797] exists,up f5e55fab-b8f1-4b4c-9e7c-53aa88ccb78f 2026-02-22T20:35:02.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:35:02.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:35:02.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:35:02.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:35:02.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:35:02.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:35:02.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:35:02.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:35:02.270 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:35:02.270 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:35:02.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:35:02.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:35:02.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:35:02.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:35:02.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:35:02.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:35:02.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:35:02.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:35:02.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:35:02.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:35:02.540 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:35:02.540 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:35:02.540 INFO:tasks.workunit.client.0.trial150.stderr:3 2026-02-22T20:35:02.540 INFO:tasks.workunit.client.0.trial150.stderr:4 2026-02-22T20:35:02.540 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:35:02.540 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:35:02.540 INFO:tasks.workunit.client.0.trial150.stderr:7 2026-02-22T20:35:02.540 INFO:tasks.workunit.client.0.trial150.stderr:8 2026-02-22T20:35:02.540 INFO:tasks.workunit.client.0.trial150.stderr:9' 2026-02-22T20:35:02.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:35:02.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:02.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:35:02.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-22T20:35:02.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-22T20:35:02.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-22T20:35:02.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:02.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:35:02.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672976 2026-02-22T20:35:02.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672976 2026-02-22T20:35:02.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976' 2026-02-22T20:35:02.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:02.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:35:02.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509455 2026-02-22T20:35:02.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509455 2026-02-22T20:35:02.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455' 2026-02-22T20:35:02.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:02.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:35:02.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710725 2026-02-22T20:35:02.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710725 2026-02-22T20:35:02.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-300647710725' 2026-02-22T20:35:02.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:02.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:35:02.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182414 2026-02-22T20:35:02.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182414 2026-02-22T20:35:02.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-300647710725 4-107374182414' 2026-02-22T20:35:02.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:02.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:35:02.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018893 2026-02-22T20:35:02.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018893 2026-02-22T20:35:02.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-300647710725 4-107374182414 5-128849018893' 2026-02-22T20:35:02.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:02.976 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:35:03.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855373 2026-02-22T20:35:03.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855373 2026-02-22T20:35:03.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-300647710725 4-107374182414 5-128849018893 6-150323855373' 2026-02-22T20:35:03.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:03.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:35:03.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691852 2026-02-22T20:35:03.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691852 2026-02-22T20:35:03.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-300647710725 4-107374182414 5-128849018893 6-150323855373 7-171798691852' 2026-02-22T20:35:03.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:03.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:35:03.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528331 2026-02-22T20:35:03.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528331 2026-02-22T20:35:03.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-300647710725 4-107374182414 5-128849018893 6-150323855373 7-171798691852 8-193273528331' 2026-02-22T20:35:03.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:03.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:35:03.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=322122547202 2026-02-22T20:35:03.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 322122547202 2026-02-22T20:35:03.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-42949672976 2-64424509455 3-300647710725 4-107374182414 5-128849018893 6-150323855373 7-171798691852 8-193273528331 9-322122547202' 2026-02-22T20:35:03.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:03.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-22T20:35:03.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:03.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:35:03.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-22T20:35:03.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:03.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-22T20:35:03.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-22T20:35:03.274 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836497 2026-02-22T20:35:03.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:03.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836497 2026-02-22T20:35:03.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:35:04.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:35:04.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:04.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836497 2026-02-22T20:35:04.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:04.597 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672976 2026-02-22T20:35:04.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:04.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:35:04.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672976 2026-02-22T20:35:04.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:04.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672976 2026-02-22T20:35:04.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2026-02-22T20:35:04.602 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672976 2026-02-22T20:35:04.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:35:04.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672976 -lt 42949672976 2026-02-22T20:35:04.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:04.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509455 2026-02-22T20:35:04.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:04.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:35:04.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509455 2026-02-22T20:35:04.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:04.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509455 2026-02-22T20:35:04.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2026-02-22T20:35:04.775 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509455 2026-02-22T20:35:04.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:35:04.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509456 -lt 64424509455 2026-02-22T20:35:04.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:04.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-300647710725 2026-02-22T20:35:04.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:35:04.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-300647710725 2026-02-22T20:35:04.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:04.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710725 2026-02-22T20:35:04.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 300647710725' 2026-02-22T20:35:04.942 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 300647710725 2026-02-22T20:35:04.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:35:05.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710725 -lt 300647710725 2026-02-22T20:35:05.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:05.109 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182414 2026-02-22T20:35:05.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:05.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:35:05.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182414 2026-02-22T20:35:05.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:05.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182414 2026-02-22T20:35:05.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2026-02-22T20:35:05.114 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.4 seq 107374182414 2026-02-22T20:35:05.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:35:05.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182414 -lt 107374182414 2026-02-22T20:35:05.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:05.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-128849018893 2026-02-22T20:35:05.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:05.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:35:05.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-128849018893 2026-02-22T20:35:05.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:05.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018893 2026-02-22T20:35:05.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 128849018893' 2026-02-22T20:35:05.283 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.5 seq 128849018893 2026-02-22T20:35:05.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:35:05.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018893 -lt 128849018893 2026-02-22T20:35:05.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:05.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855373 2026-02-22T20:35:05.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:05.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:35:05.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855373 2026-02-22T20:35:05.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:05.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855373 2026-02-22T20:35:05.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855373' 2026-02-22T20:35:05.454 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.6 seq 150323855373 2026-02-22T20:35:05.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:35:05.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855373 -lt 150323855373 2026-02-22T20:35:05.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:05.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691852 2026-02-22T20:35:05.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:05.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:35:05.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691852 2026-02-22T20:35:05.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:05.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691852 2026-02-22T20:35:05.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691852' 2026-02-22T20:35:05.627 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.7 seq 171798691852 2026-02-22T20:35:05.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:35:05.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691852 -lt 171798691852 2026-02-22T20:35:05.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:05.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528331 2026-02-22T20:35:05.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:05.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:35:05.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528331 2026-02-22T20:35:05.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:05.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528331 2026-02-22T20:35:05.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528331' 2026-02-22T20:35:05.795 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.8 seq 193273528331 2026-02-22T20:35:05.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:35:05.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528332 -lt 193273528331 2026-02-22T20:35:05.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:05.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-322122547202 2026-02-22T20:35:05.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:05.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:35:05.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-322122547202 2026-02-22T20:35:05.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:05.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=322122547202 2026-02-22T20:35:05.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 322122547202' 2026-02-22T20:35:05.964 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.9 seq 322122547202 2026-02-22T20:35:05.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:35:06.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 322122547202 -lt 322122547202 2026-02-22T20:35:06.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:35:06.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:06.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:06.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:35:06.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:06.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:06.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:06.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:06.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:06.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:06.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:06.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:35:06.472 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:06.473 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:06.473 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:06.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:35:06.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:35:06.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:35:06.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:35:06.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:35:06.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:321: corrupt_and_repair_erasure_coded: corrupt_and_repair_one td/osd-scrub-repair ecpool 5 2026-02-22T20:35:06.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-22T20:35:06.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=ecpool 2026-02-22T20:35:06.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=5 2026-02-22T20:35:06.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:35:06.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:35:06.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:35:06.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-22T20:35:06.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:35:06.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:35:06.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:35:06.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:35:06.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-22T20:35:06.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:35:06.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-22T20:35:06.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:35:06.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:35:06.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:35:06.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:35:06.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:35:07.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:35:07.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:35:07.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:35:07.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:35:07.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-22T20:35:07.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:35:07.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:35:07.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-22T20:35:07.194 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:35:07.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-22T20:35:07.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:35:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:35:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-22T20:35:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:35:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:35:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:35:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:35:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:35:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:35:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-22T20:35:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-22T20:35:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:35:07.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:35:07.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:35:07.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:35:07.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:35:07.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:35:07.296 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:35:07.296 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:35:07.296 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:35:07.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:07.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:07.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:35:07.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:35:07.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:35:07.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:35:07.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:35:07.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:35:07.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:35:07.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:35:07.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:35:07.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-22T20:35:07.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-22T20:35:07.299 INFO:tasks.workunit.client.0.trial150.stderr:start osd.5 2026-02-22T20:35:07.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:35:07.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-22T20:35:07.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-22T20:35:07.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:35:07.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:35:07.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:35:07.316 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:07.302+0000 7ff6ee179a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:07.318 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:07.306+0000 7ff6ee179a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:07.319 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:07.306+0000 7ff6ee179a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:07.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-22T20:35:07.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:35:07.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-22T20:35:07.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:35:07.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:35:07.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:07.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:35:07.454 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:35:07.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:07.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:35:07.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:35:07.742 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:07.730+0000 7ff6ee179a40 -1 Falling back to public interface 2026-02-22T20:35:08.354 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:08.342+0000 7ff6ee179a40 -1 osd.5 77 log_to_monitors true 2026-02-22T20:35:08.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:35:08.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:08.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:35:08.616 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:35:08.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:08.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:35:08.775 INFO:tasks.workunit.client.0.trial150.stderr:osd.5 up in weight 1 up_from 80 up_thru 80 down_at 77 last_clean_interval [30,77) [v2:127.0.0.1:6842/1991469186,v1:127.0.0.1:6843/1991469186] [v2:127.0.0.1:6844/1991469186,v1:127.0.0.1:6845/1991469186] exists,up 24792635-284b-494d-b483-2f02de46b8ba 2026-02-22T20:35:08.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:35:08.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:35:08.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:35:08.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:35:08.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:35:08.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:35:08.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:35:08.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:35:08.777 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:35:08.777 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:35:08.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:35:08.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:35:08.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:35:08.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:35:08.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:35:08.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:35:08.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:35:08.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:35:08.886 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:35:09.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:35:09.042 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:35:09.042 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:35:09.042 INFO:tasks.workunit.client.0.trial150.stderr:3 2026-02-22T20:35:09.042 INFO:tasks.workunit.client.0.trial150.stderr:4 2026-02-22T20:35:09.043 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:35:09.043 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:35:09.043 INFO:tasks.workunit.client.0.trial150.stderr:7 2026-02-22T20:35:09.043 INFO:tasks.workunit.client.0.trial150.stderr:8 2026-02-22T20:35:09.043 INFO:tasks.workunit.client.0.trial150.stderr:9' 2026-02-22T20:35:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:35:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:09.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:35:09.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-22T20:35:09.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-22T20:35:09.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-22T20:35:09.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:09.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:35:09.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672978 2026-02-22T20:35:09.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672978 2026-02-22T20:35:09.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672978' 2026-02-22T20:35:09.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:09.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:35:09.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509458 2026-02-22T20:35:09.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509458 2026-02-22T20:35:09.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509458' 2026-02-22T20:35:09.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:09.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:35:09.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710728 2026-02-22T20:35:09.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710728 2026-02-22T20:35:09.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509458 3-300647710728' 2026-02-22T20:35:09.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:09.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:35:09.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182416 2026-02-22T20:35:09.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182416 2026-02-22T20:35:09.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509458 3-300647710728 4-107374182416' 2026-02-22T20:35:09.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:09.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:35:09.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383682 2026-02-22T20:35:09.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383682 2026-02-22T20:35:09.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509458 3-300647710728 4-107374182416 5-343597383682' 2026-02-22T20:35:09.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:09.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:35:09.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855375 2026-02-22T20:35:09.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855375 2026-02-22T20:35:09.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509458 3-300647710728 4-107374182416 5-343597383682 6-150323855375' 2026-02-22T20:35:09.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:09.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:35:09.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691854 2026-02-22T20:35:09.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691854 2026-02-22T20:35:09.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509458 3-300647710728 4-107374182416 5-343597383682 6-150323855375 7-171798691854' 2026-02-22T20:35:09.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:09.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:35:09.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528333 2026-02-22T20:35:09.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528333 2026-02-22T20:35:09.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509458 3-300647710728 4-107374182416 5-343597383682 6-150323855375 7-171798691854 8-193273528333' 2026-02-22T20:35:09.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:09.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:35:09.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=322122547204 2026-02-22T20:35:09.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 322122547204 2026-02-22T20:35:09.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672978 2-64424509458 3-300647710728 4-107374182416 5-343597383682 6-150323855375 7-171798691854 8-193273528333 9-322122547204' 2026-02-22T20:35:09.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:09.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-22T20:35:09.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:09.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:35:09.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-22T20:35:09.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:09.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-22T20:35:09.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-22T20:35:09.776 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836499 2026-02-22T20:35:09.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:09.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836499 -lt 21474836499 2026-02-22T20:35:09.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:09.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672978 2026-02-22T20:35:09.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:09.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:35:09.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672978 2026-02-22T20:35:09.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:09.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672978 2026-02-22T20:35:09.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2026-02-22T20:35:09.951 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672978 2026-02-22T20:35:09.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:35:10.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672979 -lt 42949672978 2026-02-22T20:35:10.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:10.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509458 2026-02-22T20:35:10.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:10.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:35:10.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509458 2026-02-22T20:35:10.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:10.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509458 2026-02-22T20:35:10.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2026-02-22T20:35:10.129 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509458 2026-02-22T20:35:10.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:35:10.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509458 -lt 64424509458 2026-02-22T20:35:10.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:10.289 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-300647710728 2026-02-22T20:35:10.289 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:10.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:35:10.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-300647710728 2026-02-22T20:35:10.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:10.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710728 2026-02-22T20:35:10.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 300647710728' 2026-02-22T20:35:10.293 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 300647710728 2026-02-22T20:35:10.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:35:10.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710728 -lt 300647710728 2026-02-22T20:35:10.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:10.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182416 2026-02-22T20:35:10.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:10.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:35:10.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182416 2026-02-22T20:35:10.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:10.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182416 2026-02-22T20:35:10.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182416' 2026-02-22T20:35:10.456 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.4 seq 107374182416 2026-02-22T20:35:10.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:35:10.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182416 -lt 107374182416 2026-02-22T20:35:10.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:10.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-343597383682 2026-02-22T20:35:10.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:10.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:35:10.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-343597383682 2026-02-22T20:35:10.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:10.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383682 2026-02-22T20:35:10.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 343597383682' 2026-02-22T20:35:10.629 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.5 seq 343597383682 2026-02-22T20:35:10.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:35:10.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383682 -lt 343597383682 2026-02-22T20:35:10.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:10.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855375 2026-02-22T20:35:10.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:10.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:35:10.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855375 2026-02-22T20:35:10.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:10.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855375 2026-02-22T20:35:10.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855375' 2026-02-22T20:35:10.795 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.6 seq 150323855375 2026-02-22T20:35:10.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:35:10.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855375 -lt 150323855375 2026-02-22T20:35:10.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:10.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691854 2026-02-22T20:35:10.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:10.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:35:10.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691854 2026-02-22T20:35:10.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:10.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691854 2026-02-22T20:35:10.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691854' 2026-02-22T20:35:10.963 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.7 seq 171798691854 2026-02-22T20:35:10.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:35:11.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691854 -lt 171798691854 2026-02-22T20:35:11.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:11.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528333 2026-02-22T20:35:11.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:11.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:35:11.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528333 2026-02-22T20:35:11.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:11.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528333 2026-02-22T20:35:11.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528333' 2026-02-22T20:35:11.136 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.8 seq 193273528333 2026-02-22T20:35:11.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:35:11.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528333 -lt 193273528333 2026-02-22T20:35:11.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:11.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-322122547204 2026-02-22T20:35:11.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:11.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:35:11.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-322122547204 2026-02-22T20:35:11.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:11.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=322122547204 2026-02-22T20:35:11.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 322122547204' 2026-02-22T20:35:11.304 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.9 seq 322122547204 2026-02-22T20:35:11.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:35:11.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 322122547204 -lt 322122547204 2026-02-22T20:35:11.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:35:11.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:11.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:11.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:35:11.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:11.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:11.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:11.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:11.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:11.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:11.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:11.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:35:11.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:11.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:11.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:12.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:35:12.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:35:12.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:35:12.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg ecpool SOMETHING 2026-02-22T20:35:12.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:35:12.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:35:12.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:35:12.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:35:12.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=2.0 2026-02-22T20:35:12.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 2.0 2026-02-22T20:35:12.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:35:12.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:35:12.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:12.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:12.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:12.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:12.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:34:55.044413+0000 2026-02-22T20:35:12.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:35:12.473 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:35:12.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:34:55.044413+0000 2026-02-22T20:35:12.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:35:12.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:34:55.044413+0000 2026-02-22T20:35:12.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:35:12.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:35:12.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:12.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:12.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:12.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:12.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:12.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:12.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:34:55.044413+0000 '>' 2026-02-22T20:34:55.044413+0000 2026-02-22T20:35:12.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:13.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:13.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:13.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:13.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:13.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:13.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:13.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:13.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:34:55.044413+0000 '>' 2026-02-22T20:34:55.044413+0000 2026-02-22T20:35:13.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:14.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:14.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:14.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:14.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:14.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:14.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:14.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:14.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:34:55.044413+0000 '>' 2026-02-22T20:34:55.044413+0000 2026-02-22T20:35:14.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:15.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:15.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:15.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:15.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:15.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:15.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:15.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:16.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:13.035331+0000 '>' 2026-02-22T20:34:55.044413+0000 2026-02-22T20:35:16.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:35:16.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:35:16.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:35:16.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:35:16.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-22T20:35:16.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:35:16.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:35:16.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:35:16.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:35:16.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-22T20:35:16.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:35:16.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-22T20:35:16.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:35:16.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:35:16.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:35:16.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:35:16.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:35:16.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:35:16.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:35:16.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:35:16.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:35:16.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-22T20:35:16.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:35:16.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:35:16.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-22T20:35:16.495 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:35:16.496 INFO:tasks.workunit.client.0.trial150.stdout:hinfo_key 2026-02-22T20:35:16.496 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:35:16.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-22T20:35:16.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:35:16.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:35:16.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-22T20:35:16.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:35:16.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:35:16.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:35:16.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:35:16.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:35:16.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:35:16.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-22T20:35:16.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-22T20:35:16.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:35:16.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:35:16.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:35:16.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:35:16.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:35:16.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:35:16.596 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:35:16.596 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:35:16.596 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:35:16.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:16.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:16.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:35:16.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:35:16.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:35:16.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:35:16.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:35:16.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:35:16.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:35:16.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:35:16.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:35:16.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-22T20:35:16.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-22T20:35:16.599 INFO:tasks.workunit.client.0.trial150.stderr:start osd.9 2026-02-22T20:35:16.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:35:16.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-22T20:35:16.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-22T20:35:16.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:35:16.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:35:16.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:35:16.614 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:16.602+0000 7f0c19565a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:16.615 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:16.602+0000 7f0c19565a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:16.616 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:16.602+0000 7f0c19565a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:16.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-22T20:35:16.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:35:16.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-22T20:35:16.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:35:16.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:35:16.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:16.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:35:16.753 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:35:16.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:16.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:35:16.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:35:17.046 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:17.034+0000 7f0c19565a40 -1 Falling back to public interface 2026-02-22T20:35:17.685 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:35:17.674+0000 7f0c19565a40 -1 osd.9 81 log_to_monitors true 2026-02-22T20:35:17.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:35:17.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:17.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:35:17.921 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:35:17.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:17.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:35:18.083 INFO:tasks.workunit.client.0.trial150.stderr:osd.9 up in weight 1 up_from 85 up_thru 0 down_at 82 last_clean_interval [75,81) [v2:127.0.0.1:6874/778007663,v1:127.0.0.1:6875/778007663] [v2:127.0.0.1:6876/778007663,v1:127.0.0.1:6877/778007663] exists,up f5e55fab-b8f1-4b4c-9e7c-53aa88ccb78f 2026-02-22T20:35:18.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:35:18.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:35:18.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:35:18.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:35:18.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:35:18.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:35:18.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:35:18.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:35:18.085 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:35:18.085 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:35:18.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:35:18.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:35:18.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:35:18.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:35:18.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:35:18.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:35:18.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:35:18.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:35:18.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr:3 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr:4 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr:5 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr:6 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr:7 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr:8 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr:9' 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:18.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:35:18.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836502 2026-02-22T20:35:18.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836502 2026-02-22T20:35:18.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502' 2026-02-22T20:35:18.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:18.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:35:18.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672981 2026-02-22T20:35:18.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672981 2026-02-22T20:35:18.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-42949672981' 2026-02-22T20:35:18.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:18.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:35:18.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509461 2026-02-22T20:35:18.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509461 2026-02-22T20:35:18.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509461' 2026-02-22T20:35:18.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:18.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:35:18.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710730 2026-02-22T20:35:18.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710730 2026-02-22T20:35:18.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509461 3-300647710730' 2026-02-22T20:35:18.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:18.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:35:18.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182419 2026-02-22T20:35:18.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182419 2026-02-22T20:35:18.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509461 3-300647710730 4-107374182419' 2026-02-22T20:35:18.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:18.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:35:18.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=343597383685 2026-02-22T20:35:18.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 343597383685 2026-02-22T20:35:18.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509461 3-300647710730 4-107374182419 5-343597383685' 2026-02-22T20:35:18.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:18.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:35:18.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855378 2026-02-22T20:35:18.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855378 2026-02-22T20:35:18.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509461 3-300647710730 4-107374182419 5-343597383685 6-150323855378' 2026-02-22T20:35:18.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:18.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:35:18.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691857 2026-02-22T20:35:18.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691857 2026-02-22T20:35:18.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509461 3-300647710730 4-107374182419 5-343597383685 6-150323855378 7-171798691857' 2026-02-22T20:35:18.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:18.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:35:18.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528336 2026-02-22T20:35:18.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528336 2026-02-22T20:35:18.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509461 3-300647710730 4-107374182419 5-343597383685 6-150323855378 7-171798691857 8-193273528336' 2026-02-22T20:35:18.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:18.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:35:19.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=365072220162 2026-02-22T20:35:19.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 365072220162 2026-02-22T20:35:19.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509461 3-300647710730 4-107374182419 5-343597383685 6-150323855378 7-171798691857 8-193273528336 9-365072220162' 2026-02-22T20:35:19.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:19.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836502 2026-02-22T20:35:19.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:19.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:35:19.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836502 2026-02-22T20:35:19.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:19.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836502 2026-02-22T20:35:19.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2026-02-22T20:35:19.072 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836502 2026-02-22T20:35:19.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:19.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836501 -lt 21474836502 2026-02-22T20:35:19.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:35:20.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:35:20.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:20.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836502 2026-02-22T20:35:20.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:20.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672981 2026-02-22T20:35:20.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:20.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:35:20.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672981 2026-02-22T20:35:20.409 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:20.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672981 2026-02-22T20:35:20.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2026-02-22T20:35:20.410 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672981 2026-02-22T20:35:20.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:35:20.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672982 -lt 42949672981 2026-02-22T20:35:20.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:20.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509461 2026-02-22T20:35:20.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:20.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:35:20.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509461 2026-02-22T20:35:20.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:20.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509461 2026-02-22T20:35:20.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2026-02-22T20:35:20.577 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509461 2026-02-22T20:35:20.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:35:20.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509461 -lt 64424509461 2026-02-22T20:35:20.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:20.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-300647710730 2026-02-22T20:35:20.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:20.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:35:20.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-300647710730 2026-02-22T20:35:20.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:20.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710730 2026-02-22T20:35:20.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 300647710730' 2026-02-22T20:35:20.745 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.3 seq 300647710730 2026-02-22T20:35:20.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:35:20.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710731 -lt 300647710730 2026-02-22T20:35:20.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:20.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182419 2026-02-22T20:35:20.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:20.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:35:20.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182419 2026-02-22T20:35:20.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:20.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182419 2026-02-22T20:35:20.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182419' 2026-02-22T20:35:20.909 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.4 seq 107374182419 2026-02-22T20:35:20.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:35:21.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182419 -lt 107374182419 2026-02-22T20:35:21.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:21.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-343597383685 2026-02-22T20:35:21.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:21.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:35:21.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-343597383685 2026-02-22T20:35:21.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:21.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=343597383685 2026-02-22T20:35:21.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 343597383685' 2026-02-22T20:35:21.085 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.5 seq 343597383685 2026-02-22T20:35:21.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:35:21.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 343597383685 -lt 343597383685 2026-02-22T20:35:21.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:21.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855378 2026-02-22T20:35:21.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:21.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:35:21.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855378 2026-02-22T20:35:21.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:21.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855378 2026-02-22T20:35:21.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855378' 2026-02-22T20:35:21.248 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.6 seq 150323855378 2026-02-22T20:35:21.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:35:21.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855378 -lt 150323855378 2026-02-22T20:35:21.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:21.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691857 2026-02-22T20:35:21.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:21.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:35:21.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691857 2026-02-22T20:35:21.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:21.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691857 2026-02-22T20:35:21.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691857' 2026-02-22T20:35:21.425 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.7 seq 171798691857 2026-02-22T20:35:21.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:35:21.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691857 -lt 171798691857 2026-02-22T20:35:21.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:21.584 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528336 2026-02-22T20:35:21.584 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:21.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:35:21.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528336 2026-02-22T20:35:21.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:21.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528336 2026-02-22T20:35:21.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528336' 2026-02-22T20:35:21.588 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.8 seq 193273528336 2026-02-22T20:35:21.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:35:21.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528336 -lt 193273528336 2026-02-22T20:35:21.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:21.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-365072220162 2026-02-22T20:35:21.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:21.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:35:21.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-365072220162 2026-02-22T20:35:21.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:21.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=365072220162 2026-02-22T20:35:21.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 365072220162' 2026-02-22T20:35:21.764 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.9 seq 365072220162 2026-02-22T20:35:21.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:35:21.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 365072220162 -lt 365072220162 2026-02-22T20:35:21.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:35:21.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:21.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:22.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:35:22.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:22.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:22.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:22.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:22.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:22.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:22.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:22.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:35:22.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:22.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:22.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:22.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:35:22.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:35:22.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:35:22.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:35:22.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:35:22.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:322: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 5 9 2026-02-22T20:35:22.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-22T20:35:22.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-22T20:35:22.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=5 2026-02-22T20:35:22.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=9 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 224610"' 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 224610' 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 224611"' 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 224611' 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 224610 224611' 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 224610 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:35:22.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/224613: /' 2026-02-22T20:35:22.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING remove 2026-02-22T20:35:22.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/224615: /' 2026-02-22T20:35:23.157 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:35:23.157 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:35:23.157 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-22T20:35:23.157 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:35:23.158 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING remove 2026-02-22T20:35:23.158 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:35:23.158 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:35:23.158 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-22T20:35:23.158 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:35:23.158 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-22T20:35:23.158 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:35:23.158 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:35:23.158 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:35:23.158 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:35:23.158 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING remove 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING remove 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: remove 2#2:eb822e21:::SOMETHING:head# 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:35:23.159 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-22T20:35:23.160 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:35:23.160 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:35:23.160 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:35:23.160 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:35:23.161 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:35:23.161 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:35:23.161 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-22T20:35:23.161 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-22T20:35:23.161 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:35:23.161 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:35:23.161 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:35:23.162 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:35:23.162 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:35:23.162 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:35:23.162 INFO:tasks.workunit.client.0.trial150.stderr:224615: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:35:23.162 INFO:tasks.workunit.client.0.trial150.stderr:224615: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:35:23.162 INFO:tasks.workunit.client.0.trial150.stderr:224615: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:35:23.162 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:23.162 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:23.162 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:35:23.162 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:35:23.162 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:35:23.162 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:35:23.163 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:35:23.163 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:35:23.163 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:35:23.163 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:35:23.163 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:35:23.163 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-22T20:35:23.163 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-22T20:35:23.163 INFO:tasks.workunit.client.0.trial150.stderr:224615: start osd.9 2026-02-22T20:35:23.163 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:35:23.220 INFO:tasks.workunit.client.0.trial150.stderr:2224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:35:23.220 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:35:23.221 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-22T20:35:23.221 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:35:23.221 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:35:23.221 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:35:23.221 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:35:23.221 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-22T20:35:23.221 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:35:23.221 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-22T20:35:23.221 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:35:23.221 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:35:23.221 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:35:23.222 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:35:23.222 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:35:23.222 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:35:23.222 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:35:23.222 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:35:23.222 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:35:23.222 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-22T20:35:23.222 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:35:23.222 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:35:23.222 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-22T20:35:23.223 INFO:tasks.workunit.client.0.trial150.stderr:224613: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:35:23.223 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-22T20:35:23.223 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:35:23.223 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:35:23.223 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-22T20:35:23.223 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:35:23.223 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:35:23.223 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:35:23.223 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:35:23.224 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:35:23.224 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:35:23.224 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-22T20:35:23.224 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-22T20:35:23.224 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:35:23.225 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:35:23.225 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:35:23.225 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:35:23.225 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:35:23.225 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:35:23.225 INFO:tasks.workunit.client.0.trial150.stderr:224613: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:35:23.225 INFO:tasks.workunit.client.0.trial150.stderr:224613: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:35:23.225 INFO:tasks.workunit.client.0.trial150.stderr:224613: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:35:23.225 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:23.225 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:23.225 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:35:23.225 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:35:23.226 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:35:23.226 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:35:23.226 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:35:23.226 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:35:23.226 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:35:23.226 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:35:23.226 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:35:23.226 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-22T20:35:23.226 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-22T20:35:23.226 INFO:tasks.workunit.client.0.trial150.stderr:224613: start osd.5 2026-02-22T20:35:23.227 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: 2026-02-22T20:35:23.166+0000 7f3fc994ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: 2026-02-22T20:35:23.170+0000 7f3fc994ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: 2026-02-22T20:35:23.170+0000 7f3fc994ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: 0 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: 2026-02-22T20:35:23.666+0000 7f3fc994ea40 -1 Falling back to public interface 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: 2026-02-22T20:35:24.338+0000 7f3fc994ea40 -1 osd.9 86 log_to_monitors true 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:35:24.652 INFO:tasks.workunit.client.0.trial150.stderr:224615: 1 2026-02-22T20:35:24.653 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:24.653 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:35:24.653 INFO:tasks.workunit.client.0.trial150.stderr:224615: osd.9 up in weight 1 up_from 90 up_thru 0 down_at 87 last_clean_interval [85,86) [v2:127.0.0.1:6842/3245211049,v1:127.0.0.1:6843/3245211049] [v2:127.0.0.1:6844/3245211049,v1:127.0.0.1:6845/3245211049] exists,up f5e55fab-b8f1-4b4c-9e7c-53aa88ccb78f 2026-02-22T20:35:24.653 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:35:24.653 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:35:24.653 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:35:24.653 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:35:24.653 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:35:24.653 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:35:24.653 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:35:24.653 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:35:24.653 INFO:tasks.workunit.client.0.trial150.stderr:224615: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224615: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: ge24613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: 2026-02-22T20:35:23.226+0000 7ff48b29fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: 2026-02-22T20:35:23.226+0000 7ff48b29fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: 2026-02-22T20:35:23.226+0000 7ff48b29fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: 0 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: 2026-02-22T20:35:23.710+0000 7ff48b29fa40 -1 Falling back to public interface 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: 2026-02-22T20:35:24.390+0000 7ff48b29fa40 -1 osd.5 86 log_to_monitors true 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: 1 2026-02-22T20:35:24.708 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:24.709 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:35:24.709 INFO:tasks.workunit.client.0.trial150.stderr:224613: osd.5 up in weight 1 up_from 91 up_thru 80 down_at 87 last_clean_interval [80,86) [v2:127.0.0.1:6874/1281488722,v1:127.0.0.1:6875/1281488722] [v2:127.0.0.1:6876/1281488722,v1:127.0.0.1:6877/1281488722] exists,up 24792635-284b-494d-b483-2f02de46b8ba 2026-02-22T20:35:24.709 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:35:24.709 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:35:24.709 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:35:24.709 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:35:24.709 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:35:24.709 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:35:24.709 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:35:24.709 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:35:24.709 INFO:tasks.workunit.client.0.trial150.stderr:224613: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:35:25.208 INFO:tasks.workunit.client.0.trial150.stderr:224613: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: gt_timeout_delays: echo true 2026-02-22T20:35:25.208 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:35:25.208 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:35:25.208 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:35:25.208 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:35:25.208 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:35:25.209 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:35:25.209 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:35:25.209 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:35:25.209 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:35:25.209 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:35:25.209 INFO:tasks.workunit.client.0.trial150.stderr:224615: 1 2026-02-22T20:35:25.209 INFO:tasks.workunit.client.0.trial150.stderr:224615: 2 2026-02-22T20:35:25.209 INFO:tasks.workunit.client.0.trial150.stderr:224615: 3 2026-02-22T20:35:25.209 INFO:tasks.workunit.client.0.trial150.stderr:224615: 4 2026-02-22T20:35:25.209 INFO:tasks.workunit.client.0.trial150.stderr:224615: 5 2026-02-22T20:35:25.209 INFO:tasks.workunit.client.0.trial150.stderr:224615: 6 2026-02-22T20:35:25.210 INFO:tasks.workunit.client.0.trial150.stderr:224615: 7 2026-02-22T20:35:25.210 INFO:tasks.workunit.client.0.trial150.stderr:224615: 8 2026-02-22T20:35:25.210 INFO:tasks.workunit.client.0.trial150.stderr:224615: 9' 2026-02-22T20:35:25.210 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:35:25.210 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.210 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:35:25.210 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836504 2026-02-22T20:35:25.210 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836504 2026-02-22T20:35:25.210 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504' 2026-02-22T20:35:25.210 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.211 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:35:25.211 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672984 2026-02-22T20:35:25.211 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672984 2026-02-22T20:35:25.211 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-42949672984' 2026-02-22T20:35:25.211 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.211 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:35:25.211 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509463 2026-02-22T20:35:25.211 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509463 2026-02-22T20:35:25.211 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509463' 2026-02-22T20:35:25.211 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.211 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:35:25.212 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710733 2026-02-22T20:35:25.212 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710733 2026-02-22T20:35:25.212 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509463 3-300647710733' 2026-02-22T20:35:25.212 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: et_timeout_delays: echo true 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: 1 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: 2 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: 3 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: 4 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: 5 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: 6 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: 7 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: 8 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: 9' 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836505 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836505 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505' 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672985 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672985 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985' 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509464 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509464 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464' 2026-02-22T20:35:25.270 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.271 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:35:25.271 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710734 2026-02-22T20:35:25.271 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710734 2026-02-22T20:35:25.271 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-300647710734' 2026-02-22T20:35:25.271 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.641 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258:flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:35:25.641 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182421 2026-02-22T20:35:25.641 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182421 2026-02-22T20:35:25.641 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509463 3-300647710733 4-107374182421' 2026-02-22T20:35:25.641 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=390842023938 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 390842023938 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509463 3-300647710733 4-107374182421 5-390842023938' 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855380 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855380 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509463 3-300647710733 4-107374182421 5-390842023938 6-150323855380' 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691859 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691859 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509463 3-300647710733 4-107374182421 5-390842023938 6-150323855380 7-171798691859' 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528339 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528339 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509463 3-300647710733 4-107374182421 5-390842023938 6-150323855380 7-171798691859 8-193273528339' 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056642 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056642 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836504 1-42949672984 2-64424509463 3-300647710733 4-107374182421 5-390842023938 6-150323855380 7-171798691859 8-193273528339 9-386547056642' 2026-02-22T20:35:25.642 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:25.711 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubu flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:35:25.712 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182422 2026-02-22T20:35:25.712 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182422 2026-02-22T20:35:25.712 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-300647710734 4-107374182422' 2026-02-22T20:35:25.712 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.712 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:35:25.712 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=390842023939 2026-02-22T20:35:25.712 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 390842023939 2026-02-22T20:35:25.712 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-300647710734 4-107374182422 5-390842023939' 2026-02-22T20:35:25.712 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.712 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:35:25.712 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855381 2026-02-22T20:35:25.713 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855381 2026-02-22T20:35:25.713 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-300647710734 4-107374182422 5-390842023939 6-150323855381' 2026-02-22T20:35:25.713 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.713 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:35:25.713 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691860 2026-02-22T20:35:25.713 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691860 2026-02-22T20:35:25.713 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-300647710734 4-107374182422 5-390842023939 6-150323855381 7-171798691860' 2026-02-22T20:35:25.713 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.713 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:35:25.713 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528340 2026-02-22T20:35:25.713 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528340 2026-02-22T20:35:25.713 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-300647710734 4-107374182422 5-390842023939 6-150323855381 7-171798691860 8-193273528340' 2026-02-22T20:35:25.714 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:25.714 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:35:25.714 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056643 2026-02-22T20:35:25.714 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056643 2026-02-22T20:35:25.715 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-42949672985 2-64424509464 3-300647710734 4-107374182422 5-390842023939 6-150323855381 7-171798691860 8-193273528340 9-386547056643' 2026-02-22T20:35:25.715 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:27.319 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836504 2026-02-22T20:35:27.319 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.319 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:35:27.319 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836504 2026-02-22T20:35:27.319 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:27.319 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836504 2026-02-22T20:35:27.319 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2026-02-22T20:35:27.319 INFO:tasks.workunit.client.0.trial150.stderr:224615: waiting osd.0 seq 21474836504 2026-02-22T20:35:27.319 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:27.320 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836504 2026-02-22T20:35:27.320 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:35:27.320 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:35:27.320 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:27.320 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836506 -lt 21474836504 2026-02-22T20:35:27.320 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:27.320 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672984 2026-02-22T20:35:27.320 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.320 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:35:27.320 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672984 2026-02-22T20:35:27.320 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:27.320 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672984 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: waiting osd.1 seq 42949672984 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672985 -lt 42949672984 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509463 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509463 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509463 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2026-02-22T20:35:27.321 INFO:tasks.workunit.client.0.trial150.stderr:224615: waiting osd.2 seq 64424509463 2026-02-22T20:35:27.322 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:35:27.322 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509464 -lt 64424509463 2026-02-22T20:35:27.359 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_untu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836505 2026-02-22T20:35:27.359 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.359 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:35:27.359 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836505 2026-02-22T20:35:27.359 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:27.359 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836505 2026-02-22T20:35:27.360 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2026-02-22T20:35:27.360 INFO:tasks.workunit.client.0.trial150.stderr:224613: waiting osd.0 seq 21474836505 2026-02-22T20:35:27.360 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:27.360 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836503 -lt 21474836505 2026-02-22T20:35:27.360 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:35:27.360 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:35:27.360 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:27.360 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836506 -lt 21474836505 2026-02-22T20:35:27.360 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:27.360 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672985 2026-02-22T20:35:27.360 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.361 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:35:27.361 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672985 2026-02-22T20:35:27.361 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:27.361 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672985 2026-02-22T20:35:27.361 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2026-02-22T20:35:27.361 INFO:tasks.workunit.client.0.trial150.stderr:224613: waiting osd.1 seq 42949672985 2026-02-22T20:35:27.361 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:35:27.361 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672985 -lt 42949672985 2026-02-22T20:35:27.361 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:27.361 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509464 2026-02-22T20:35:27.361 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.361 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:35:27.362 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509464 2026-02-22T20:35:27.362 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:27.362 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509464 2026-02-22T20:35:27.362 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2026-02-22T20:35:27.362 INFO:tasks.workunit.client.0.trial150.stderr:224613: waiting osd.2 seq 64424509464 2026-02-22T20:35:27.362 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:35:27.362 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509464 -lt 64424509464 2026-02-22T20:35:27.821 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flushpg_stats: for s in $seqs 2026-02-22T20:35:27.821 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-300647710733 2026-02-22T20:35:27.821 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.821 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:35:27.821 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-300647710733 2026-02-22T20:35:27.821 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:27.821 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710733 2026-02-22T20:35:27.821 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 300647710733' 2026-02-22T20:35:27.821 INFO:tasks.workunit.client.0.trial150.stderr:224615: waiting osd.3 seq 300647710733 2026-02-22T20:35:27.822 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:35:27.822 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710734 -lt 300647710733 2026-02-22T20:35:27.822 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:27.822 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182421 2026-02-22T20:35:27.822 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.822 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:35:27.822 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182421 2026-02-22T20:35:27.822 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:27.822 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182421 2026-02-22T20:35:27.822 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182421' 2026-02-22T20:35:27.822 INFO:tasks.workunit.client.0.trial150.stderr:224615: waiting osd.4 seq 107374182421 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182423 -lt 107374182421 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-390842023938 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-390842023938 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=390842023938 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 390842023938' 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: waiting osd.5 seq 390842023938 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:35:27.823 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 390842023939 -lt 390842023938 2026-02-22T20:35:27.824 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:27.824 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855380 2026-02-22T20:35:27.824 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.824 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:35:27.866 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/s_pg_stats: for s in $seqs 2026-02-22T20:35:27.866 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-300647710734 2026-02-22T20:35:27.866 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.866 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:35:27.866 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-300647710734 2026-02-22T20:35:27.866 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:27.866 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710734 2026-02-22T20:35:27.867 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 300647710734' 2026-02-22T20:35:27.867 INFO:tasks.workunit.client.0.trial150.stderr:224613: waiting osd.3 seq 300647710734 2026-02-22T20:35:27.867 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:35:27.867 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710734 -lt 300647710734 2026-02-22T20:35:27.867 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:27.867 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182422 2026-02-22T20:35:27.867 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.867 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:35:27.867 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182422 2026-02-22T20:35:27.867 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:27.867 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182422 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182422' 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: waiting osd.4 seq 107374182422 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182423 -lt 107374182422 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-390842023939 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-390842023939 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=390842023939 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 390842023939' 2026-02-22T20:35:27.868 INFO:tasks.workunit.client.0.trial150.stderr:224613: waiting osd.5 seq 390842023939 2026-02-22T20:35:27.869 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:35:27.869 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 390842023939 -lt 390842023939 2026-02-22T20:35:27.869 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:27.869 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855381 2026-02-22T20:35:27.869 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:27.869 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:35:28.323 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/tandalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855380 2026-02-22T20:35:28.323 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855380 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855380' 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: waiting osd.6 seq 150323855380 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855382 -lt 150323855380 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691859 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691859 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691859 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691859' 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: waiting osd.7 seq 171798691859 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691861 -lt 171798691859 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528339 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528339 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528339 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528339' 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: waiting osd.8 seq 193273528339 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528340 -lt 193273528339 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:28.324 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-386547056642 2026-02-22T20:35:28.325 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:28.325 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:35:28.325 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-386547056642 2026-02-22T20:35:28.325 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:28.325 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056642 2026-02-22T20:35:28.366 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osdstandalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855381 2026-02-22T20:35:28.367 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:28.367 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855381 2026-02-22T20:35:28.367 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855381' 2026-02-22T20:35:28.367 INFO:tasks.workunit.client.0.trial150.stderr:224613: waiting osd.6 seq 150323855381 2026-02-22T20:35:28.367 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:35:28.367 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855382 -lt 150323855381 2026-02-22T20:35:28.367 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:28.368 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691860 2026-02-22T20:35:28.368 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:28.368 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:35:28.368 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691860 2026-02-22T20:35:28.368 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:28.368 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691860 2026-02-22T20:35:28.368 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691860' 2026-02-22T20:35:28.368 INFO:tasks.workunit.client.0.trial150.stderr:224613: waiting osd.7 seq 171798691860 2026-02-22T20:35:28.369 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:35:28.369 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691861 -lt 171798691860 2026-02-22T20:35:28.369 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:28.369 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528340 2026-02-22T20:35:28.369 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:28.369 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:35:28.369 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528340 2026-02-22T20:35:28.370 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:28.370 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528340 2026-02-22T20:35:28.370 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528340' 2026-02-22T20:35:28.370 INFO:tasks.workunit.client.0.trial150.stderr:224613: waiting osd.8 seq 193273528340 2026-02-22T20:35:28.370 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:35:28.370 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528340 -lt 193273528340 2026-02-22T20:35:28.370 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:28.370 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-386547056643 2026-02-22T20:35:28.371 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:28.371 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:35:28.371 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-386547056643 2026-02-22T20:35:28.371 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:28.371 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056643 2026-02-22T20:35:29.034 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting os.9 seq 386547056642' 2026-02-22T20:35:29.034 INFO:tasks.workunit.client.0.trial150.stderr:224615: waiting osd.9 seq 386547056642 2026-02-22T20:35:29.034 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:35:29.034 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056643 -lt 386547056642 2026-02-22T20:35:29.034 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:35:29.034 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:29.034 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:29.034 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:35:29.034 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:29.034 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:29.035 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:29.035 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:29.035 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:29.035 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:29.035 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:29.035 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:35:29.035 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:29.035 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:29.035 INFO:tasks.workunit.client.0.trial150.stderr:224615: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:29.036 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:35:29.036 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:35:29.036 INFO:tasks.workunit.client.0.trial150.stderr:224615: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:35:29.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:35:29.087 INFO:tasks.workunit.client.0.trial150.stderr:d.9 seq 386547056643' 2026-02-22T20:35:29.087 INFO:tasks.workunit.client.0.trial150.stderr:224613: waiting osd.9 seq 386547056643 2026-02-22T20:35:29.087 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:35:29.087 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056643 -lt 386547056643 2026-02-22T20:35:29.088 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:35:29.088 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:29.088 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:29.088 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:35:29.088 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:29.088 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:29.088 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:29.088 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:29.088 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:29.088 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:29.088 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:29.088 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:35:29.089 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:29.089 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:29.089 INFO:tasks.workunit.client.0.trial150.stderr:224613: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:29.089 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:35:29.089 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:35:29.089 INFO:tasks.workunit.client.0.trial150.stderr:224613: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:35:29.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:35:29.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:35:29.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 224611 2026-02-22T20:35:29.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:35:29.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:35:29.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:35:29.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-22T20:35:29.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:35:29.090 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-22T20:35:29.090 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:35:29.090 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:35:29.090 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:35:29.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:35:29.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-22T20:35:29.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-22T20:35:29.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:35:29.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:35:29.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:29.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:29.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:29.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:29.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:35:13.035331+0000 2026-02-22T20:35:29.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:35:29.555 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:35:29.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:35:13.035331+0000 2026-02-22T20:35:29.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:35:29.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:35:13.035331+0000 2026-02-22T20:35:29.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:35:29.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:35:29.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:29.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:29.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:29.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:29.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:29.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:29.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:13.035331+0000 '>' 2026-02-22T20:35:13.035331+0000 2026-02-22T20:35:29.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:30.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:30.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:30.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:30.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:30.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:30.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:30.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:30.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:13.035331+0000 '>' 2026-02-22T20:35:13.035331+0000 2026-02-22T20:35:30.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:31.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:31.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:31.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:31.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:31.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:31.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:31.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:32.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:13.035331+0000 '>' 2026-02-22T20:35:13.035331+0000 2026-02-22T20:35:32.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:33.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:33.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:33.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:33.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:33.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:33.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:33.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:33.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:13.035331+0000 '>' 2026-02-22T20:35:13.035331+0000 2026-02-22T20:35:33.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:34.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:34.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:34.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:34.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:34.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:34.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:34.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:34.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:13.035331+0000 '>' 2026-02-22T20:35:13.035331+0000 2026-02-22T20:35:34.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:35.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:35.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:35.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:35.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:35.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:35.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:35.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:35.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:13.035331+0000 '>' 2026-02-22T20:35:13.035331+0000 2026-02-22T20:35:35.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:36.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:36.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:36.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:36.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:36.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:36.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:36.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:36.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:30.174324+0000 '>' 2026-02-22T20:35:13.035331+0000 2026-02-22T20:35:36.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:35:36.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-22T20:35:36.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:35:36.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:35:36.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:35:36.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 228072"' 2026-02-22T20:35:36.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 228072' 2026-02-22T20:35:36.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:35:36.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:35:36.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:35:36.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 228073"' 2026-02-22T20:35:36.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 228073' 2026-02-22T20:35:36.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-22T20:35:36.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 228072 228073' 2026-02-22T20:35:36.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:35:36.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:35:36.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 228072 2026-02-22T20:35:36.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:35:36.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/228075: /' 2026-02-22T20:35:36.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:35:36.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/228077: /' 2026-02-22T20:35:37.181 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:35:37.181 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:35:37.181 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=9 2026-02-22T20:35:37.181 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:35:37.181 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:35:37.181 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:35:37.181 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:35:37.181 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=9 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.9 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 9 SOMETHING list-attrs 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=9 2026-02-22T20:35:37.182 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:35:37.183 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:35:37.183 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/9 SOMETHING list-attrs 2026-02-22T20:35:37.183 INFO:tasks.workunit.client.0.trial150.stderr:228077: _ 2026-02-22T20:35:37.183 INFO:tasks.workunit.client.0.trial150.stderr:228077: hinfo_key 2026-02-22T20:35:37.183 INFO:tasks.workunit.client.0.trial150.stderr:228077: snapset 2026-02-22T20:35:37.183 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 9 2026-02-22T20:35:37.183 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:35:37.183 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:35:37.183 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=9 2026-02-22T20:35:37.183 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:35:37.183 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/9 2026-02-22T20:35:37.183 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:35:37.184 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:35:37.184 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:35:37.184 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:35:37.184 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/9' 2026-02-22T20:35:37.184 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/9/journal' 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:35:37.185 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:35:37.186 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:35:37.186 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:35:37.186 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:35:37.186 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:35:37.186 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:35:37.186 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:35:37.186 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:35:37.186 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/9 2026-02-22T20:35:37.186 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.9 2026-02-22T20:35:37.186 INFO:tasks.workunit.client.0.trial150.stderr:228077: start osd.9 2026-02-22T20:35:37.201 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 9 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/9 --osd-journal=td/osd-scrub-repair/9/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:35:37.201 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:35:37.201 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-22T20:35:37.201 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:35:37.201 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:35:37.201 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:35:37.201 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:35:37.201 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-22T20:35:37.201 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:35:37.202 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:35:37.203 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2026-02-22T20:35:37.203 INFO:tasks.workunit.client.0.trial150.stderr:228075: _ 2026-02-22T20:35:37.203 INFO:tasks.workunit.client.0.trial150.stderr:228075: hinfo_key 2026-02-22T20:35:37.203 INFO:tasks.workunit.client.0.trial150.stderr:228075: snapset 2026-02-22T20:35:37.203 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-22T20:35:37.203 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:35:37.203 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:35:37.203 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-22T20:35:37.203 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:35:37.203 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:35:37.203 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:35:37.203 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:35:37.204 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:35:37.204 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:35:37.204 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-22T20:35:37.204 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-22T20:35:37.204 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:35:37.204 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:35:37.204 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:35:37.205 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:35:37.206 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:35:37.206 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:35:37.206 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:35:37.206 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:35:37.206 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-22T20:35:37.206 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-22T20:35:37.206 INFO:tasks.workunit.client.0.trial150.stderr:228075: start osd.5 2026-02-22T20:35:38.672 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:35:38.672 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/9/whoami 2026-02-22T20:35:38.672 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 9 = 9 ']' 2026-02-22T20:35:38.672 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:35:38.672 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:35:38.672 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: 2026-02-22T20:35:37.190+0000 7f795259ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: 2026-02-22T20:35:37.194+0000 7f795259ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: 2026-02-22T20:35:37.194+0000 7f795259ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 9 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: 0 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:38.673 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:35:38.674 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:35:38.674 INFO:tasks.workunit.client.0.trial150.stderr:228077: 2026-02-22T20:35:37.678+0000 7f795259ba40 -1 Falling back to public interface 2026-02-22T20:35:38.674 INFO:tasks.workunit.client.0.trial150.stderr:228077: 2026-02-22T20:35:38.434+0000 7f795259ba40 -1 osd.9 92 log_to_monitors true 2026-02-22T20:35:38.674 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:35:38.674 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:38.674 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:35:38.674 INFO:tasks.workunit.client.0.trial150.stderr:228077: 1 2026-02-22T20:35:38.674 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:38.674 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-22T20:35:38.674 INFO:tasks.workunit.client.0.trial150.stderr:228077: osd.9 up in weight 1 up_from 96 up_thru 0 down_at 93 last_clean_interval [90,92) [v2:127.0.0.1:6842/978033274,v1:127.0.0.1:6843/978033274] [v2:127.0.0.1:6844/978033274,v1:127.0.0.1:6845/978033274] exists,up f5e55fab-b8f1-4b4c-9e7c-53aa88ccb78f 2026-02-22T20:35:38.674 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:35:38.674 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:35:38.675 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:35:38.675 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:35:38.675 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:35:38.675 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:35:38.675 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:35:38.675 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:35:38.675 INFO:tasks.workunit.client.0.trial150.stderr:228077: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:35:38.687 INFO:tasks.workunit.client.0.trial150.stderr:228077: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:15very_ops 2026-02-22T20:35:38.687 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-22T20:35:38.687 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-22T20:35:38.687 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:35:38.687 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:35:38.687 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:35:38.688 INFO:tasks.workunit.client.0.trial150.stderr:228075: 2026-02-22T20:35:37.206+0000 7f8e3e2c2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:38.688 INFO:tasks.workunit.client.0.trial150.stderr:228075: 2026-02-22T20:35:37.210+0000 7f8e3e2c2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:38.688 INFO:tasks.workunit.client.0.trial150.stderr:228075: 2026-02-22T20:35:37.210+0000 7f8e3e2c2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:38.688 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-22T20:35:38.688 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:35:38.688 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-22T20:35:38.688 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:35:38.688 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:35:38.689 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:38.689 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:35:38.689 INFO:tasks.workunit.client.0.trial150.stderr:228075: 0 2026-02-22T20:35:38.689 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:38.689 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:35:38.689 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:35:38.689 INFO:tasks.workunit.client.0.trial150.stderr:228075: 2026-02-22T20:35:37.710+0000 7f8e3e2c2a40 -1 Falling back to public interface 2026-02-22T20:35:38.689 INFO:tasks.workunit.client.0.trial150.stderr:228075: 2026-02-22T20:35:38.458+0000 7f8e3e2c2a40 -1 osd.5 92 log_to_monitors true 2026-02-22T20:35:38.690 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:35:38.690 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:38.690 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:35:38.690 INFO:tasks.workunit.client.0.trial150.stderr:228075: 1 2026-02-22T20:35:38.690 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:38.690 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:35:38.690 INFO:tasks.workunit.client.0.trial150.stderr:228075: osd.5 up in weight 1 up_from 96 up_thru 91 down_at 93 last_clean_interval [91,92) [v2:127.0.0.1:6874/137415268,v1:127.0.0.1:6875/137415268] [v2:127.0.0.1:6876/137415268,v1:127.0.0.1:6877/137415268] exists,up 24792635-284b-494d-b483-2f02de46b8ba 2026-02-22T20:35:38.690 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:35:38.690 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:35:38.691 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:35:38.691 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:35:38.691 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:35:38.691 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:35:38.691 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:35:38.691 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:35:38.691 INFO:tasks.workunit.client.0.trial150.stderr:228075: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228075: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:193: get_timeout_delays: echo true 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: 1 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: 2 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: 3 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: 4 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: 5 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: 6 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: 7 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: 8 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: 9' 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836509 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836509 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509' 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672988 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672988 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672988' 2026-02-22T20:35:39.234 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.235 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:35:39.235 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509468 2026-02-22T20:35:39.235 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509468 2026-02-22T20:35:39.235 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468' 2026-02-22T20:35:39.235 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.235 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:35:39.235 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710738 2026-02-22T20:35:39.235 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710738 2026-02-22T20:35:39.235 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-300647710738' 2026-02-22T20:35:39.235 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.241 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:593: get_timeout_delays: echo true 2026-02-22T20:35:39.241 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:35:39.241 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:35:39.241 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:35:39.241 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:35:39.241 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:35:39.242 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:35:39.242 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:35:39.242 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:35:39.242 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:35:39.242 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:35:39.242 INFO:tasks.workunit.client.0.trial150.stderr:228075: 1 2026-02-22T20:35:39.242 INFO:tasks.workunit.client.0.trial150.stderr:228075: 2 2026-02-22T20:35:39.242 INFO:tasks.workunit.client.0.trial150.stderr:228075: 3 2026-02-22T20:35:39.243 INFO:tasks.workunit.client.0.trial150.stderr:228075: 4 2026-02-22T20:35:39.243 INFO:tasks.workunit.client.0.trial150.stderr:228075: 5 2026-02-22T20:35:39.243 INFO:tasks.workunit.client.0.trial150.stderr:228075: 6 2026-02-22T20:35:39.243 INFO:tasks.workunit.client.0.trial150.stderr:228075: 7 2026-02-22T20:35:39.243 INFO:tasks.workunit.client.0.trial150.stderr:228075: 8 2026-02-22T20:35:39.243 INFO:tasks.workunit.client.0.trial150.stderr:228075: 9' 2026-02-22T20:35:39.243 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:35:39.244 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.244 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:35:39.244 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836510 2026-02-22T20:35:39.244 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836510 2026-02-22T20:35:39.244 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510' 2026-02-22T20:35:39.244 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.245 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:35:39.245 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672989 2026-02-22T20:35:39.245 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672989 2026-02-22T20:35:39.245 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989' 2026-02-22T20:35:39.245 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.245 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:35:39.245 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509469 2026-02-22T20:35:39.245 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509469 2026-02-22T20:35:39.246 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509469' 2026-02-22T20:35:39.246 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.246 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:35:39.246 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=300647710739 2026-02-22T20:35:39.246 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 300647710739 2026-02-22T20:35:39.246 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509469 3-300647710739' 2026-02-22T20:35:39.246 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.665 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:35:39.665 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182426 2026-02-22T20:35:39.665 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182426 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-300647710738 4-107374182426' 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860418 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860418 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-300647710738 4-107374182426 5-412316860418' 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855385 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855385 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-300647710738 4-107374182426 5-412316860418 6-150323855385' 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691864 2026-02-22T20:35:39.666 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691864 2026-02-22T20:35:39.667 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-300647710738 4-107374182426 5-412316860418 6-150323855385 7-171798691864' 2026-02-22T20:35:39.667 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.667 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:35:39.667 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528343 2026-02-22T20:35:39.667 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528343 2026-02-22T20:35:39.667 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-300647710738 4-107374182426 5-412316860418 6-150323855385 7-171798691864 8-193273528343' 2026-02-22T20:35:39.667 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.667 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:35:39.667 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860418 2026-02-22T20:35:39.667 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860418 2026-02-22T20:35:39.667 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672988 2-64424509468 3-300647710738 4-107374182426 5-412316860418 6-150323855385 7-171798691864 8-193273528343 9-412316860418' 2026-02-22T20:35:39.667 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:39.678 INFO:tasks.workunit.client.0.trial150.stderr:228077: //ho:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:35:39.678 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182427 2026-02-22T20:35:39.678 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182427 2026-02-22T20:35:39.678 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509469 3-300647710739 4-107374182427' 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860419 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860419 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509469 3-300647710739 4-107374182427 5-412316860419' 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855386 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855386 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509469 3-300647710739 4-107374182427 5-412316860419 6-150323855386' 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691865 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691865 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509469 3-300647710739 4-107374182427 5-412316860419 6-150323855386 7-171798691865' 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528344 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528344 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509469 3-300647710739 4-107374182427 5-412316860419 6-150323855386 7-171798691865 8-193273528344' 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860419 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860419 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836510 1-42949672989 2-64424509469 3-300647710739 4-107374182427 5-412316860419 6-150323855386 7-171798691865 8-193273528344 9-412316860419' 2026-02-22T20:35:39.679 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:41.325 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836510 2026-02-22T20:35:41.326 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.326 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:35:41.326 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836510 2026-02-22T20:35:41.326 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:41.326 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836510 2026-02-22T20:35:41.326 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2026-02-22T20:35:41.326 INFO:tasks.workunit.client.0.trial150.stderr:228075: waiting osd.0 seq 21474836510 2026-02-22T20:35:41.326 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:41.326 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836510 2026-02-22T20:35:41.326 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:35:41.326 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:35:41.327 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:41.327 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836510 2026-02-22T20:35:41.327 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:41.327 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672989 2026-02-22T20:35:41.327 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.327 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:35:41.327 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672989 2026-02-22T20:35:41.327 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:41.327 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672989 2026-02-22T20:35:41.327 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2026-02-22T20:35:41.327 INFO:tasks.workunit.client.0.trial150.stderr:228075: waiting osd.1 seq 42949672989 2026-02-22T20:35:41.327 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:35:41.328 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672990 -lt 42949672989 2026-02-22T20:35:41.328 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:41.328 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509469 2026-02-22T20:35:41.328 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.328 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:35:41.328 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509469 2026-02-22T20:35:41.328 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:41.328 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509469 2026-02-22T20:35:41.328 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2026-02-22T20:35:41.328 INFO:tasks.workunit.client.0.trial150.stderr:228075: waiting osd.2 seq 64424509469 2026-02-22T20:35:41.328 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:35:41.329 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509469 -lt 64424509469 2026-02-22T20:35:41.338 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266:me/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836509 2026-02-22T20:35:41.338 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.338 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:35:41.338 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836509 2026-02-22T20:35:41.338 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:41.338 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836509 2026-02-22T20:35:41.338 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2026-02-22T20:35:41.338 INFO:tasks.workunit.client.0.trial150.stderr:228077: waiting osd.0 seq 21474836509 2026-02-22T20:35:41.338 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836508 -lt 21474836509 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836509 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672988 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672988 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672988 2026-02-22T20:35:41.339 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2026-02-22T20:35:41.340 INFO:tasks.workunit.client.0.trial150.stderr:228077: waiting osd.1 seq 42949672988 2026-02-22T20:35:41.340 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:35:41.340 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672990 -lt 42949672988 2026-02-22T20:35:41.340 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:41.340 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509468 2026-02-22T20:35:41.340 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.340 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:35:41.340 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509468 2026-02-22T20:35:41.340 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:41.340 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509468 2026-02-22T20:35:41.340 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2026-02-22T20:35:41.340 INFO:tasks.workunit.client.0.trial150.stderr:228077: waiting osd.2 seq 64424509468 2026-02-22T20:35:41.341 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:35:41.341 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509469 -lt 64424509468 2026-02-22T20:35:41.813 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:41.814 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-300647710739 2026-02-22T20:35:41.814 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.814 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:35:41.814 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-300647710739 2026-02-22T20:35:41.814 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:41.814 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710739 2026-02-22T20:35:41.814 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 300647710739' 2026-02-22T20:35:41.815 INFO:tasks.workunit.client.0.trial150.stderr:228075: waiting osd.3 seq 300647710739 2026-02-22T20:35:41.815 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:35:41.815 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710739 -lt 300647710739 2026-02-22T20:35:41.815 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:41.815 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182427 2026-02-22T20:35:41.815 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.815 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:35:41.815 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182427 2026-02-22T20:35:41.815 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:41.815 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182427 2026-02-22T20:35:41.815 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182427' 2026-02-22T20:35:41.815 INFO:tasks.workunit.client.0.trial150.stderr:228075: waiting osd.4 seq 107374182427 2026-02-22T20:35:41.816 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:35:41.816 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182427 -lt 107374182427 2026-02-22T20:35:41.816 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:41.816 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-412316860419 2026-02-22T20:35:41.816 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.816 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:35:41.816 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-412316860419 2026-02-22T20:35:41.816 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:41.816 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860419 2026-02-22T20:35:41.816 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 412316860419' 2026-02-22T20:35:41.816 INFO:tasks.workunit.client.0.trial150.stderr:228075: waiting osd.5 seq 412316860419 2026-02-22T20:35:41.816 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:35:41.817 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860419 -lt 412316860419 2026-02-22T20:35:41.817 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:41.817 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855386 2026-02-22T20:35:41.817 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.817 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:35:41.831 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.clientflush_pg_stats: for s in $seqs 2026-02-22T20:35:41.831 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-300647710738 2026-02-22T20:35:41.831 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.831 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:35:41.831 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-300647710738 2026-02-22T20:35:41.831 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=300647710738 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 300647710738' 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: waiting osd.3 seq 300647710738 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 300647710739 -lt 300647710738 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182426 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182426 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182426 2026-02-22T20:35:41.832 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182426' 2026-02-22T20:35:41.833 INFO:tasks.workunit.client.0.trial150.stderr:228077: waiting osd.4 seq 107374182426 2026-02-22T20:35:41.833 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:35:41.833 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182427 -lt 107374182426 2026-02-22T20:35:41.833 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:41.833 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-412316860418 2026-02-22T20:35:41.833 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.833 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:35:41.833 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-412316860418 2026-02-22T20:35:41.833 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:41.833 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860418 2026-02-22T20:35:41.833 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 412316860418' 2026-02-22T20:35:41.833 INFO:tasks.workunit.client.0.trial150.stderr:228077: waiting osd.5 seq 412316860418 2026-02-22T20:35:41.834 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:35:41.834 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860419 -lt 412316860418 2026-02-22T20:35:41.834 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:41.834 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855385 2026-02-22T20:35:41.834 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:41.834 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:35:42.304 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client..0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855386 2026-02-22T20:35:42.305 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:42.305 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855386 2026-02-22T20:35:42.305 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855386' 2026-02-22T20:35:42.305 INFO:tasks.workunit.client.0.trial150.stderr:228075: waiting osd.6 seq 150323855386 2026-02-22T20:35:42.305 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:35:42.305 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855386 -lt 150323855386 2026-02-22T20:35:42.305 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:42.305 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691865 2026-02-22T20:35:42.305 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:42.305 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:35:42.305 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691865 2026-02-22T20:35:42.306 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:42.306 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691865 2026-02-22T20:35:42.306 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691865' 2026-02-22T20:35:42.306 INFO:tasks.workunit.client.0.trial150.stderr:228075: waiting osd.7 seq 171798691865 2026-02-22T20:35:42.306 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:35:42.306 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691866 -lt 171798691865 2026-02-22T20:35:42.306 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:42.306 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528344 2026-02-22T20:35:42.306 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:42.306 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:35:42.306 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528344 2026-02-22T20:35:42.306 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:42.307 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528344 2026-02-22T20:35:42.307 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528344' 2026-02-22T20:35:42.307 INFO:tasks.workunit.client.0.trial150.stderr:228075: waiting osd.8 seq 193273528344 2026-02-22T20:35:42.307 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:35:42.307 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528345 -lt 193273528344 2026-02-22T20:35:42.307 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:42.307 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-412316860419 2026-02-22T20:35:42.307 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:42.307 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:35:42.307 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-412316860419 2026-02-22T20:35:42.307 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:42.307 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860419 2026-02-22T20:35:42.344 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'wait0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855385 2026-02-22T20:35:42.344 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:42.344 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855385 2026-02-22T20:35:42.345 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855385' 2026-02-22T20:35:42.345 INFO:tasks.workunit.client.0.trial150.stderr:228077: waiting osd.6 seq 150323855385 2026-02-22T20:35:42.345 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:35:42.345 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855386 -lt 150323855385 2026-02-22T20:35:42.345 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:42.345 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691864 2026-02-22T20:35:42.345 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:42.345 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:35:42.345 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691864 2026-02-22T20:35:42.345 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:42.345 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691864 2026-02-22T20:35:42.345 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691864' 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: waiting osd.7 seq 171798691864 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691866 -lt 171798691864 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528343 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528343 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528343 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528343' 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: waiting osd.8 seq 193273528343 2026-02-22T20:35:42.346 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:35:42.347 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528345 -lt 193273528343 2026-02-22T20:35:42.347 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:42.347 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-412316860418 2026-02-22T20:35:42.347 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:42.347 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:35:42.347 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-412316860418 2026-02-22T20:35:42.347 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:42.347 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860418 2026-02-22T20:35:43.285 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waitiing osd.9 seq 412316860419' 2026-02-22T20:35:43.285 INFO:tasks.workunit.client.0.trial150.stderr:228075: waiting osd.9 seq 412316860419 2026-02-22T20:35:43.285 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:35:43.285 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860419 -lt 412316860419 2026-02-22T20:35:43.285 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:35:43.285 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:43.285 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:43.286 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:35:43.286 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:43.286 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:43.286 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:43.286 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:43.286 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:43.286 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:43.286 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:43.286 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:35:43.286 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:43.286 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:43.287 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:43.287 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:35:43.287 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-22T20:35:43.287 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:35:43.287 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-22T20:35:43.287 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:35:43.287 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:35:43.287 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:35:43.287 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:43.287 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:43.287 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:43.287 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:43.288 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:43.288 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:43.288 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:43.323 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cng osd.9 seq 412316860418' 2026-02-22T20:35:43.323 INFO:tasks.workunit.client.0.trial150.stderr:228077: waiting osd.9 seq 412316860418 2026-02-22T20:35:43.323 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:35:43.323 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860419 -lt 412316860418 2026-02-22T20:35:43.323 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:35:43.323 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:43.323 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:43.323 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:35:43.323 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:43.323 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:35:43.324 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-22T20:35:43.325 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:35:43.325 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:35:43.325 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:35:43.325 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:43.325 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:43.325 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:43.325 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:43.325 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:43.325 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:43.325 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:44.223 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/ceephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:35:44.224 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:44.224 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:44.224 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:44.224 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:35:44.224 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-22T20:35:44.224 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:35:44.224 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:35:44.224 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:35:44.224 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:35:44.225 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:35:44.225 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:35:44.225 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:35:44.225 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:35:44.225 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:35:44.225 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:35:44.225 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:35:44.225 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:35:44.225 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:35:44.225 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:44.225 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:44.225 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:44.226 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:44.226 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:44.226 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:44.226 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:44.226 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:35:44.226 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:44.226 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:44.226 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:44.226 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:35:44.267 INFO:tasks.workunit.client.0.trial150.stderr:228075:phtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:35:44.268 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:44.268 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:44.268 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:44.268 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:35:44.268 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-22T20:35:44.268 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:35:44.268 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:35:44.268 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:35:44.268 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:35:44.269 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:35:44.269 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:35:44.269 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:35:44.269 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:35:44.269 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:35:44.269 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:35:44.269 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:35:44.269 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:35:44.269 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:35:44.269 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:44.269 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:44.269 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:44.270 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:44.270 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:44.270 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:44.270 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:44.270 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:35:44.270 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:44.270 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:44.270 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:44.270 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:35:45.172 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-22T20:35:45.172 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:35:45.172 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:35:45.172 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:35:45.172 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:35:45.172 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:35:45.172 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:35:45.173 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:35:45.173 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:35:45.173 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:35:45.173 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-22T20:35:45.173 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:35:45.173 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-22T20:35:45.173 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:35:45.173 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:45.173 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:45.173 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:45.174 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:45.174 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:45.174 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:45.174 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:45.174 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:35:45.174 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:45.174 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:45.174 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:45.174 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:35:45.174 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-22T20:35:45.174 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:35:45.174 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:35:45.175 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:35:45.208 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-22T20:35:45.208 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:35:45.208 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:35:45.208 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:35:45.208 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:35:45.208 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:35:45.208 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:35:45.208 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:35:45.209 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:35:45.209 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:35:45.209 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-22T20:35:45.209 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:35:45.209 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-22T20:35:45.209 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:35:45.209 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:45.209 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:45.209 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:45.209 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:45.209 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:45.209 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:45.210 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:45.210 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:35:45.210 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:45.210 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:45.210 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:45.210 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:35:45.210 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-22T20:35:45.210 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:35:45.210 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:35:45.210 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:35:46.529 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu//cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:35:46.530 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:35:46.530 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:35:46.530 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:35:46.530 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:35:46.530 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:35:46.530 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-22T20:35:46.530 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:35:46.530 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-22T20:35:46.531 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:35:46.531 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:46.531 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:46.531 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:46.531 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:46.531 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:46.531 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:46.531 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:46.531 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:35:46.531 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:46.531 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:46.532 INFO:tasks.workunit.client.0.trial150.stderr:228075: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:46.532 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:35:46.532 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:35:46.532 INFO:tasks.workunit.client.0.trial150.stderr:228075: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:35:46.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:35:46.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:35:46.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 228073 2026-02-22T20:35:46.539 INFO:tasks.workunit.client.0.trial150.stderr:cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:35:46.539 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:35:46.539 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:35:46.539 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:35:46.539 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:35:46.540 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:35:46.540 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-22T20:35:46.540 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:35:46.540 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-22T20:35:46.540 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:35:46.540 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:46.540 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:46.540 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:46.540 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:46.540 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:46.541 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:46.541 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:46.541 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:35:46.541 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:46.541 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:46.541 INFO:tasks.workunit.client.0.trial150.stderr:228077: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:46.541 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:35:46.541 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:35:46.541 INFO:tasks.workunit.client.0.trial150.stderr:228077: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:35:46.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:35:46.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:35:46.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:35:46.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:35:46.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-22T20:35:46.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:35:46.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:35:46.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:35:46.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:323: corrupt_and_repair_erasure_coded: corrupt_and_repair_two td/osd-scrub-repair ecpool 3 5 2026-02-22T20:35:46.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:245: corrupt_and_repair_two: local dir=td/osd-scrub-repair 2026-02-22T20:35:46.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:246: corrupt_and_repair_two: local poolname=ecpool 2026-02-22T20:35:46.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:247: corrupt_and_repair_two: local first=3 2026-02-22T20:35:46.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:248: corrupt_and_repair_two: local second=5 2026-02-22T20:35:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:253: corrupt_and_repair_two: pids= 2026-02-22T20:35:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:254: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:35:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:35:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:35:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 231481"' 2026-02-22T20:35:46.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 231481' 2026-02-22T20:35:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:255: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:35:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:35:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:35:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 231482"' 2026-02-22T20:35:46.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 231482' 2026-02-22T20:35:46.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:256: corrupt_and_repair_two: wait_background pids 2026-02-22T20:35:46.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 231481 231482' 2026-02-22T20:35:46.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:35:46.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:35:46.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 231481 2026-02-22T20:35:46.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:35:46.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/231484: /' 2026-02-22T20:35:46.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:35:46.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/231486: /' 2026-02-22T20:35:47.169 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:35:47.169 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:35:47.169 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-22T20:35:47.170 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:35:47.170 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:35:47.170 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:35:47.170 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:35:47.170 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-22T20:35:47.170 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:35:47.170 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-22T20:35:47.170 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:35:47.170 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:35:47.170 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:35:47.170 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING remove 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING remove 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: remove 1#2:eb822e21:::SOMETHING:head# 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:35:47.171 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:35:47.172 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-22T20:35:47.172 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:35:47.172 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:35:47.172 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:35:47.172 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:35:47.173 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:35:47.173 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:35:47.173 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-22T20:35:47.173 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-22T20:35:47.173 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:35:47.174 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:35:47.175 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:35:47.175 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:35:47.175 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:35:47.175 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:35:47.175 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:35:47.175 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:35:47.175 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-22T20:35:47.175 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-22T20:35:47.175 INFO:tasks.workunit.client.0.trial150.stderr:231486: start osd.5 2026-02-22T20:35:47.175 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:35:47.208 INFO:tasks.workunit.client.0.trial150.stderr:2231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:35:47.208 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:35:47.208 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:35:47.208 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:35:47.209 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: remove 0#2:eb822e21:::SOMETHING:head# 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:35:47.210 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:35:47.211 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:35:47.211 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:35:47.212 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:35:47.212 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:35:47.212 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:35:47.212 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:35:47.212 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:35:47.212 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:35:47.212 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:35:47.212 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:35:47.212 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:35:47.212 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:35:47.212 INFO:tasks.workunit.client.0.trial150.stderr:231484: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:35:47.212 INFO:tasks.workunit.client.0.trial150.stderr:231484: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:35:47.213 INFO:tasks.workunit.client.0.trial150.stderr:231484: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:35:47.213 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:47.213 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:35:47.213 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:35:47.213 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:35:47.213 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:35:47.213 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:35:47.213 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:35:47.213 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:35:47.213 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:35:47.213 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:35:47.213 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:35:47.214 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:35:47.214 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:35:47.214 INFO:tasks.workunit.client.0.trial150.stderr:231484: start osd.3 2026-02-22T20:35:47.214 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:35:48.663 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-22T20:35:48.664 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-22T20:35:48.664 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:35:48.664 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:35:48.664 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:35:48.664 INFO:tasks.workunit.client.0.trial150.stderr:231486: 2026-02-22T20:35:47.182+0000 7f943a695a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:48.664 INFO:tasks.workunit.client.0.trial150.stderr:231486: 2026-02-22T20:35:47.186+0000 7f943a695a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:48.664 INFO:tasks.workunit.client.0.trial150.stderr:231486: 2026-02-22T20:35:47.186+0000 7f943a695a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:48.664 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-22T20:35:48.664 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:35:48.664 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: 0 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: 2026-02-22T20:35:47.682+0000 7f943a695a40 -1 Falling back to public interface 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: 2026-02-22T20:35:48.434+0000 7f943a695a40 -1 osd.5 97 log_to_monitors true 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:48.665 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: 1 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: osd.5 up in weight 1 up_from 101 up_thru 96 down_at 98 last_clean_interval [96,97) [v2:127.0.0.1:6826/1172420203,v1:127.0.0.1:6827/1172420203] [v2:127.0.0.1:6828/1172420203,v1:127.0.0.1:6829/1172420203] exists,up 24792635-284b-494d-b483-2f02de46b8ba 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:35:48.666 INFO:tasks.workunit.client.0.trial150.stderr:231486: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:35:48.697 INFO:tasks.workunit.client.0.trial150.stderr:231486: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: 31484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:35:48.698 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:35:48.698 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:35:48.698 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:35:48.698 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:35:48.698 INFO:tasks.workunit.client.0.trial150.stderr:231484: 2026-02-22T20:35:47.214+0000 7f8eaf737a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:48.698 INFO:tasks.workunit.client.0.trial150.stderr:231484: 2026-02-22T20:35:47.214+0000 7f8eaf737a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:48.698 INFO:tasks.workunit.client.0.trial150.stderr:231484: 2026-02-22T20:35:47.214+0000 7f8eaf737a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:35:48.698 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:35:48.698 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:35:48.698 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:35:48.699 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:35:48.699 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:35:48.699 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:48.699 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:35:48.699 INFO:tasks.workunit.client.0.trial150.stderr:231484: 0 2026-02-22T20:35:48.699 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:48.699 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:35:48.699 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:35:48.699 INFO:tasks.workunit.client.0.trial150.stderr:231484: 2026-02-22T20:35:47.706+0000 7f8eaf737a40 -1 Falling back to public interface 2026-02-22T20:35:48.699 INFO:tasks.workunit.client.0.trial150.stderr:231484: 2026-02-22T20:35:48.462+0000 7f8eaf737a40 -1 osd.3 97 log_to_monitors true 2026-02-22T20:35:48.699 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:35:48.699 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: 1 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: osd.3 up in weight 1 up_from 101 up_thru 96 down_at 98 last_clean_interval [70,97) [v2:127.0.0.1:6874/867056675,v1:127.0.0.1:6875/867056675] [v2:127.0.0.1:6876/867056675,v1:127.0.0.1:6877/867056675] exists,up e04d4bb4-34bb-453b-844d-af8ddea2016c 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:35:48.700 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:35:48.701 INFO:tasks.workunit.client.0.trial150.stderr:231484: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:35:49.222 INFO:tasks.workunit.client.0.trial150.stderr:231484: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_get_timeout_delays: echo true 2026-02-22T20:35:49.222 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:35:49.222 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:35:49.222 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:35:49.222 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:35:49.222 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:35:49.222 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:35:49.222 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:35:49.222 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: 1 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: 2 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: 3 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: 4 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: 5 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: 6 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: 7 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: 8 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: 9' 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:35:49.223 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.224 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:35:49.224 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836513 2026-02-22T20:35:49.224 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836513 2026-02-22T20:35:49.224 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513' 2026-02-22T20:35:49.224 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.224 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:35:49.224 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672992 2026-02-22T20:35:49.224 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672992 2026-02-22T20:35:49.224 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672992' 2026-02-22T20:35:49.224 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.224 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:35:49.224 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509472 2026-02-22T20:35:49.225 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509472 2026-02-22T20:35:49.225 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-64424509472' 2026-02-22T20:35:49.225 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.225 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:35:49.225 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696898 2026-02-22T20:35:49.225 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696898 2026-02-22T20:35:49.225 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-64424509472 3-433791696898' 2026-02-22T20:35:49.225 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.250 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258timeout_delays: echo true 2026-02-22T20:35:49.250 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:35:49.251 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:35:49.251 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:35:49.251 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:35:49.251 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:35:49.251 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:35:49.251 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:35:49.251 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:35:49.251 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:35:49.251 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:35:49.251 INFO:tasks.workunit.client.0.trial150.stderr:231484: 1 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: 2 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: 3 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: 4 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: 5 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: 6 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: 7 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: 8 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: 9' 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836514 2026-02-22T20:35:49.252 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836514 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514' 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672993 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672993 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993' 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509473 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509473 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509473' 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.253 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:35:49.254 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696899 2026-02-22T20:35:49.254 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696899 2026-02-22T20:35:49.254 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509473 3-433791696899' 2026-02-22T20:35:49.254 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.661 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: fl: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182430 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182430 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-64424509472 3-433791696898 4-107374182430' 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696898 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696898 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-64424509472 3-433791696898 4-107374182430 5-433791696898' 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855389 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855389 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-64424509472 3-433791696898 4-107374182430 5-433791696898 6-150323855389' 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691868 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691868 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-64424509472 3-433791696898 4-107374182430 5-433791696898 6-150323855389 7-171798691868' 2026-02-22T20:35:49.662 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.663 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:35:49.663 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528347 2026-02-22T20:35:49.663 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528347 2026-02-22T20:35:49.663 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-64424509472 3-433791696898 4-107374182430 5-433791696898 6-150323855389 7-171798691868 8-193273528347' 2026-02-22T20:35:49.663 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.663 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:35:49.663 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860422 2026-02-22T20:35:49.663 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860422 2026-02-22T20:35:49.663 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672992 2-64424509472 3-433791696898 4-107374182430 5-433791696898 6-150323855389 7-171798691868 8-193273528347 9-412316860422' 2026-02-22T20:35:49.663 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:49.689 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/uush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:35:49.689 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182431 2026-02-22T20:35:49.689 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182431 2026-02-22T20:35:49.689 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509473 3-433791696899 4-107374182431' 2026-02-22T20:35:49.689 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.689 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:35:49.689 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696899 2026-02-22T20:35:49.689 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696899 2026-02-22T20:35:49.689 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509473 3-433791696899 4-107374182431 5-433791696899' 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855390 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855390 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509473 3-433791696899 4-107374182431 5-433791696899 6-150323855390' 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691869 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691869 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509473 3-433791696899 4-107374182431 5-433791696899 6-150323855390 7-171798691869' 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528348 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528348 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509473 3-433791696899 4-107374182431 5-433791696899 6-150323855390 7-171798691869 8-193273528348' 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860423 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860423 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-42949672993 2-64424509473 3-433791696899 4-107374182431 5-433791696899 6-150323855390 7-171798691869 8-193273528348 9-412316860423' 2026-02-22T20:35:49.690 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntbuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836513 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836513 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836513 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: waiting osd.0 seq 21474836513 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836513 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836514 -lt 21474836513 2026-02-22T20:35:51.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672992 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672992 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672992 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: waiting osd.1 seq 42949672992 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672994 -lt 42949672992 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509472 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509472 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509472 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: waiting osd.2 seq 64424509472 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:35:51.317 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509473 -lt 64424509472 2026-02-22T20:35:51.342 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flusu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836514 2026-02-22T20:35:51.342 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.342 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:35:51.342 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836514 2026-02-22T20:35:51.342 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:51.342 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836514 2026-02-22T20:35:51.343 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2026-02-22T20:35:51.343 INFO:tasks.workunit.client.0.trial150.stderr:231484: waiting osd.0 seq 21474836514 2026-02-22T20:35:51.343 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:51.343 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836514 2026-02-22T20:35:51.343 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:35:51.343 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:35:51.343 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:35:51.343 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836514 -lt 21474836514 2026-02-22T20:35:51.343 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:51.343 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672993 2026-02-22T20:35:51.344 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.344 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:35:51.344 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672993 2026-02-22T20:35:51.344 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:51.344 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672993 2026-02-22T20:35:51.344 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2026-02-22T20:35:51.344 INFO:tasks.workunit.client.0.trial150.stderr:231484: waiting osd.1 seq 42949672993 2026-02-22T20:35:51.344 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:35:51.344 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672994 -lt 42949672993 2026-02-22T20:35:51.344 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:51.344 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509473 2026-02-22T20:35:51.345 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.345 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:35:51.345 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509473 2026-02-22T20:35:51.345 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:51.345 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509473 2026-02-22T20:35:51.345 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509473' 2026-02-22T20:35:51.345 INFO:tasks.workunit.client.0.trial150.stderr:231484: waiting osd.2 seq 64424509473 2026-02-22T20:35:51.345 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:35:51.345 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509473 -lt 64424509473 2026-02-22T20:35:51.818 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pgh_pg_stats: for s in $seqs 2026-02-22T20:35:51.818 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-433791696898 2026-02-22T20:35:51.818 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-433791696898 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696898 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 433791696898' 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: waiting osd.3 seq 433791696898 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696898 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182430 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182430 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182430 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182430' 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: waiting osd.4 seq 107374182430 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182431 -lt 107374182430 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-433791696898 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:35:51.819 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-433791696898 2026-02-22T20:35:51.820 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:51.820 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696898 2026-02-22T20:35:51.820 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 433791696898' 2026-02-22T20:35:51.820 INFO:tasks.workunit.client.0.trial150.stderr:231486: waiting osd.5 seq 433791696898 2026-02-22T20:35:51.820 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:35:51.820 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696898 2026-02-22T20:35:51.820 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:51.820 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855389 2026-02-22T20:35:51.820 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.820 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:35:51.846 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa_stats: for s in $seqs 2026-02-22T20:35:51.846 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-433791696899 2026-02-22T20:35:51.846 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.846 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:35:51.846 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-433791696899 2026-02-22T20:35:51.846 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:51.846 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696899 2026-02-22T20:35:51.846 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 433791696899' 2026-02-22T20:35:51.846 INFO:tasks.workunit.client.0.trial150.stderr:231484: waiting osd.3 seq 433791696899 2026-02-22T20:35:51.846 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696899 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182431 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182431 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182431 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182431' 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: waiting osd.4 seq 107374182431 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182431 -lt 107374182431 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:51.847 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-433791696899 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-433791696899 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696899 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 433791696899' 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: waiting osd.5 seq 433791696899 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696899 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855390 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:51.848 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:35:52.314 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/sta/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855389 2026-02-22T20:35:52.314 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:52.314 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855389 2026-02-22T20:35:52.314 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855389' 2026-02-22T20:35:52.314 INFO:tasks.workunit.client.0.trial150.stderr:231486: waiting osd.6 seq 150323855389 2026-02-22T20:35:52.314 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855390 -lt 150323855389 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691868 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691868 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691868 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691868' 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: waiting osd.7 seq 171798691868 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691870 -lt 171798691868 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528347 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528347 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528347 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528347' 2026-02-22T20:35:52.315 INFO:tasks.workunit.client.0.trial150.stderr:231486: waiting osd.8 seq 193273528347 2026-02-22T20:35:52.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:35:52.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528349 -lt 193273528347 2026-02-22T20:35:52.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:52.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-412316860422 2026-02-22T20:35:52.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:52.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:35:52.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-412316860422 2026-02-22T20:35:52.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:52.316 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860422 2026-02-22T20:35:52.352 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting ondalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855390 2026-02-22T20:35:52.352 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:52.352 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855390 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855390' 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: waiting osd.6 seq 150323855390 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855390 -lt 150323855390 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691869 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691869 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691869 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691869' 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: waiting osd.7 seq 171798691869 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:35:52.353 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691870 -lt 171798691869 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528348 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528348 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528348 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528348' 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: waiting osd.8 seq 193273528348 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528349 -lt 193273528348 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-412316860423 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-412316860423 2026-02-22T20:35:52.354 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:35:52.355 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860423 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9sd.9 seq 412316860422' 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: waiting osd.9 seq 412316860422 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860423 -lt 412316860422 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-22T20:35:53.295 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:35:53.296 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-22T20:35:53.296 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:35:53.296 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:35:53.296 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:35:53.296 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:53.296 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:53.296 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:53.296 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:53.296 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:53.296 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:53.296 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephte seq 412316860423' 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: waiting osd.9 seq 412316860423 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860423 -lt 412316860423 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:53.323 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:53.324 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:54.258 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/cst/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:35:54.258 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:54.258 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:54.258 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:54.258 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:35:54.259 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-22T20:35:54.259 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:35:54.259 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:35:54.259 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:35:54.259 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:35:54.259 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:35:54.259 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:35:54.259 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:35:54.259 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:35:54.259 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:35:54.259 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:35:54.260 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:35:54.260 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:35:54.260 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:35:54.260 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:54.260 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:54.260 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:54.260 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:54.260 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:54.260 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:54.260 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:54.260 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:35:54.260 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:54.261 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:54.261 INFO:tasks.workunit.client.0.trial150.stderr:231486: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:54.261 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:35:54.261 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:35:54.261 INFO:tasks.workunit.client.0.trial150.stderr:231486: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:35:54.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:35:54.263 INFO:tasks.workunit.client.0.trial150.stderr:lone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:35:54.263 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:54.263 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:54.263 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:54.263 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:35:54.263 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-22T20:35:54.263 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:35:54.263 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:35:54.264 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:231484: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:231484: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:35:54.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:35:54.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:35:54.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 231482 2026-02-22T20:35:54.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:35:54.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:35:54.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:35:54.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:257: corrupt_and_repair_two: return_code=0 2026-02-22T20:35:54.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:258: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:35:54.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: get_pg ecpool SOMETHING 2026-02-22T20:35:54.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:35:54.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:35:54.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:35:54.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:35:54.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:263: corrupt_and_repair_two: local pg=2.0 2026-02-22T20:35:54.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:264: corrupt_and_repair_two: repair 2.0 2026-02-22T20:35:54.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:35:54.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:35:54.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:54.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:54.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:54.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:54.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:35:30.174324+0000 2026-02-22T20:35:54.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:35:54.731 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-22T20:35:54.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:35:30.174324+0000 2026-02-22T20:35:54.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:35:54.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:35:30.174324+0000 2026-02-22T20:35:54.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:35:54.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:35:54.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:54.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:54.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:54.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:54.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:54.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:54.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:30.174324+0000 '>' 2026-02-22T20:35:30.174324+0000 2026-02-22T20:35:54.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:55.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:55.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:55.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:55.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:55.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:55.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:55.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:56.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:30.174324+0000 '>' 2026-02-22T20:35:30.174324+0000 2026-02-22T20:35:56.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:57.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:57.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:57.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:57.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:57.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:57.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:57.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:57.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:30.174324+0000 '>' 2026-02-22T20:35:30.174324+0000 2026-02-22T20:35:57.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:58.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:58.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:58.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:58.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:58.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:58.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:58.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:58.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:30.174324+0000 '>' 2026-02-22T20:35:30.174324+0000 2026-02-22T20:35:58.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:35:59.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:35:59.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:35:59.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:35:59.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:35:59.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:35:59.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:35:59.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:35:59.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:30.174324+0000 '>' 2026-02-22T20:35:30.174324+0000 2026-02-22T20:35:59.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:36:00.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:36:00.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:36:00.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:36:00.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:36:00.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:00.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:00.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:36:00.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:35:55.534106+0000 '>' 2026-02-22T20:35:30.174324+0000 2026-02-22T20:36:00.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:36:00.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:268: corrupt_and_repair_two: pids= 2026-02-22T20:36:00.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:269: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:36:00.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:36:00.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:36:00.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 235162"' 2026-02-22T20:36:00.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 235162' 2026-02-22T20:36:00.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:270: corrupt_and_repair_two: run_in_background pids objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:36:00.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:36:00.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:36:00.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 235163"' 2026-02-22T20:36:00.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 235163' 2026-02-22T20:36:00.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:271: corrupt_and_repair_two: wait_background pids 2026-02-22T20:36:00.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 235162 235163' 2026-02-22T20:36:00.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:36:00.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:36:00.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 235162 2026-02-22T20:36:00.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:36:00.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/235165: /' 2026-02-22T20:36:00.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:36:00.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/235167: /' 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=5 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=5 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.5 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:36:01.164 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 5 SOMETHING list-attrs 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=5 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/5 SOMETHING list-attrs 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: _ 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: hinfo_key 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: snapset 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 5 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=5 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/5 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:36:01.165 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/5' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/5/journal' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/5 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.5 2026-02-22T20:36:01.168 INFO:tasks.workunit.client.0.trial150.stderr:235167: start osd.5 2026-02-22T20:36:01.181 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 5 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/5 --osd-journal=td/osd-scrub-repair/5/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_reco235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:36:01.181 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:36:01.181 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:36:01.181 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:36:01.181 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:36:01.181 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:36:01.181 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:36:01.182 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:36:01.182 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:36:01.182 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:36:01.182 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:36:01.182 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:36:01.182 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:36:01.182 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:36:01.182 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:36:01.182 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:36:01.182 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING list-attrs 2026-02-22T20:36:01.182 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:36:01.183 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:36:01.183 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:36:01.183 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:36:01.183 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:36:01.183 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING list-attrs 2026-02-22T20:36:01.183 INFO:tasks.workunit.client.0.trial150.stderr:235165: Error getting attr on : 2.0s0_head,0#-4:00000000:::deep_scrub_2.0s0:head#, (61) No data available 2026-02-22T20:36:01.183 INFO:tasks.workunit.client.0.trial150.stderr:235165: Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-22T20:36:01.183 INFO:tasks.workunit.client.0.trial150.stderr:235165: _ 2026-02-22T20:36:01.183 INFO:tasks.workunit.client.0.trial150.stderr:235165: hinfo_key 2026-02-22T20:36:01.183 INFO:tasks.workunit.client.0.trial150.stderr:235165: snapset 2026-02-22T20:36:01.183 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:36:01.184 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:36:01.184 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:36:01.184 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:36:01.184 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:36:01.184 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:36:01.184 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:36:01.185 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:36:01.185 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:36:01.185 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:36:01.185 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:36:01.185 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:36:01.185 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:36:01.185 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:36:01.185 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:36:01.185 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:36:01.185 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:01.185 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:01.185 INFO:tasks.workunit.client.0.trial150.stderr:235165: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:01.186 INFO:tasks.workunit.client.0.trial150.stderr:235165: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:01.186 INFO:tasks.workunit.client.0.trial150.stderr:235165: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:01.186 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:01.186 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:01.186 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:36:01.186 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:36:01.186 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:36:01.186 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:36:01.186 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:36:01.186 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:36:01.186 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:36:01.187 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:36:01.187 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:36:01.187 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:36:01.187 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:36:01.187 INFO:tasks.workunit.client.0.trial150.stderr:235165: start osd.3 2026-02-22T20:36:02.654 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.very_ops 2026-02-22T20:36:02.654 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/5/whoami 2026-02-22T20:36:02.654 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 5 = 5 ']' 2026-02-22T20:36:02.655 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:36:02.655 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:36:02.655 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:36:02.655 INFO:tasks.workunit.client.0.trial150.stderr:235167: 2026-02-22T20:36:01.170+0000 7fc7e1ce6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:02.655 INFO:tasks.workunit.client.0.trial150.stderr:235167: 2026-02-22T20:36:01.174+0000 7fc7e1ce6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:02.655 INFO:tasks.workunit.client.0.trial150.stderr:235167: 2026-02-22T20:36:01.174+0000 7fc7e1ce6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:02.655 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 5 2026-02-22T20:36:02.655 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:36:02.655 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-22T20:36:02.655 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:36:02.655 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: 0 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: 2026-02-22T20:36:01.638+0000 7fc7e1ce6a40 -1 Falling back to public interface 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: 2026-02-22T20:36:02.362+0000 7fc7e1ce6a40 -1 osd.5 102 log_to_monitors true 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: 1 2026-02-22T20:36:02.656 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:02.657 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-22T20:36:02.657 INFO:tasks.workunit.client.0.trial150.stderr:235167: osd.5 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [101,102) [v2:127.0.0.1:6826/1779207290,v1:127.0.0.1:6827/1779207290] [v2:127.0.0.1:6828/1779207290,v1:127.0.0.1:6829/1779207290] exists,up 24792635-284b-494d-b483-2f02de46b8ba 2026-02-22T20:36:02.657 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:36:02.657 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:36:02.657 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:36:02.657 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:36:02.657 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:36:02.657 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:36:02.657 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:36:02.657 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:36:02.657 INFO:tasks.workunit.client.0.trial150.stderr:235167: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:36:02.660 INFO:tasks.workunit.client.0.trial150.stderr:235167: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helog' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:36:02.661 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:36:02.661 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:36:02.661 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:36:02.661 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:36:02.661 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:36:02.661 INFO:tasks.workunit.client.0.trial150.stderr:235165: 2026-02-22T20:36:01.190+0000 7fc899a92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:02.661 INFO:tasks.workunit.client.0.trial150.stderr:235165: 2026-02-22T20:36:01.190+0000 7fc899a92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:02.661 INFO:tasks.workunit.client.0.trial150.stderr:235165: 2026-02-22T20:36:01.194+0000 7fc899a92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:02.661 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:36:02.662 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:36:02.662 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:36:02.662 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:36:02.662 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:36:02.662 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:02.662 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:36:02.662 INFO:tasks.workunit.client.0.trial150.stderr:235165: 0 2026-02-22T20:36:02.662 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:02.662 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:36:02.662 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:02.662 INFO:tasks.workunit.client.0.trial150.stderr:235165: 2026-02-22T20:36:01.678+0000 7fc899a92a40 -1 Falling back to public interface 2026-02-22T20:36:02.662 INFO:tasks.workunit.client.0.trial150.stderr:235165: 2026-02-22T20:36:02.410+0000 7fc899a92a40 -1 osd.3 102 log_to_monitors true 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: 1 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: osd.3 up in weight 1 up_from 107 up_thru 101 down_at 103 last_clean_interval [101,102) [v2:127.0.0.1:6874/2743203308,v1:127.0.0.1:6875/2743203308] [v2:127.0.0.1:6876/2743203308,v1:127.0.0.1:6877/2743203308] exists,up e04d4bb4-34bb-453b-844d-af8ddea2016c 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:36:02.663 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:36:02.664 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:36:03.217 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:36:03.218 INFO:tasks.workunit.client.0.trial150.stderr:235165: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:36:03.218 INFO:tasks.workunit.client.0.trial150.stderr:235165: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:36:03.218 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:36:03.218 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:36:03.218 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:36:03.218 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:36:03.218 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:36:03.218 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:36:03.219 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:36:03.219 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:36:03.219 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:36:03.219 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:36:03.219 INFO:tasks.workunit.client.0.trial150.stderr:235165: 1 2026-02-22T20:36:03.219 INFO:tasks.workunit.client.0.trial150.stderr:235165: 2 2026-02-22T20:36:03.219 INFO:tasks.workunit.client.0.trial150.stderr:235165: 3 2026-02-22T20:36:03.219 INFO:tasks.workunit.client.0.trial150.stderr:235165: 4 2026-02-22T20:36:03.219 INFO:tasks.workunit.client.0.trial150.stderr:235165: 5 2026-02-22T20:36:03.219 INFO:tasks.workunit.client.0.trial150.stderr:235165: 6 2026-02-22T20:36:03.220 INFO:tasks.workunit.client.0.trial150.stderr:235165: 7 2026-02-22T20:36:03.220 INFO:tasks.workunit.client.0.trial150.stderr:235165: 8 2026-02-22T20:36:03.220 INFO:tasks.workunit.client.0.trial150.stderr:235165: 9' 2026-02-22T20:36:03.220 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:36:03.220 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.220 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:36:03.220 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836519 2026-02-22T20:36:03.220 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836519 2026-02-22T20:36:03.220 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519' 2026-02-22T20:36:03.220 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.220 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:36:03.221 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672998 2026-02-22T20:36:03.221 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672998 2026-02-22T20:36:03.221 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998' 2026-02-22T20:36:03.221 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.221 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:36:03.221 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509478 2026-02-22T20:36:03.221 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509478 2026-02-22T20:36:03.221 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478' 2026-02-22T20:36:03.221 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.221 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:36:03.221 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=459561500675 2026-02-22T20:36:03.222 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 459561500675 2026-02-22T20:36:03.222 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-lpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:36:03.222 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:36:03.222 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:36:03.222 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:36:03.222 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:36:03.222 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:36:03.222 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:36:03.222 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:36:03.222 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:36:03.222 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:36:03.222 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:36:03.223 INFO:tasks.workunit.client.0.trial150.stderr:235167: 1 2026-02-22T20:36:03.223 INFO:tasks.workunit.client.0.trial150.stderr:235167: 2 2026-02-22T20:36:03.223 INFO:tasks.workunit.client.0.trial150.stderr:235167: 3 2026-02-22T20:36:03.223 INFO:tasks.workunit.client.0.trial150.stderr:235167: 4 2026-02-22T20:36:03.223 INFO:tasks.workunit.client.0.trial150.stderr:235167: 5 2026-02-22T20:36:03.223 INFO:tasks.workunit.client.0.trial150.stderr:235167: 6 2026-02-22T20:36:03.223 INFO:tasks.workunit.client.0.trial150.stderr:235167: 7 2026-02-22T20:36:03.223 INFO:tasks.workunit.client.0.trial150.stderr:235167: 8 2026-02-22T20:36:03.223 INFO:tasks.workunit.client.0.trial150.stderr:235167: 9' 2026-02-22T20:36:03.223 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:36:03.224 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.224 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:36:03.224 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836518 2026-02-22T20:36:03.224 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836518 2026-02-22T20:36:03.224 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518' 2026-02-22T20:36:03.224 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.224 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:36:03.224 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672997 2026-02-22T20:36:03.224 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672997 2026-02-22T20:36:03.224 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997' 2026-02-22T20:36:03.224 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.225 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:36:03.225 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509477 2026-02-22T20:36:03.225 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509477 2026-02-22T20:36:03.225 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477' 2026-02-22T20:36:03.225 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.225 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:36:03.225 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=459561500674 2026-02-22T20:36:03.225 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 459561500674 2026-02-22T20:36:03.225 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-459561500674' 2026-02-22T20:36:03.225 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.653 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-64424509478 3-459561500675' 2026-02-22T20:36:03.653 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.654 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:36:03.654 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182435 2026-02-22T20:36:03.654 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182435 2026-02-22T20:36:03.654 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-459561500675 4-107374182435' 2026-02-22T20:36:03.654 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.654 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:36:03.654 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533379 2026-02-22T20:36:03.654 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533379 2026-02-22T20:36:03.654 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-459561500675 4-107374182435 5-455266533379' 2026-02-22T20:36:03.654 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.655 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:36:03.655 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855394 2026-02-22T20:36:03.655 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855394 2026-02-22T20:36:03.655 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-459561500675 4-107374182435 5-455266533379 6-150323855394' 2026-02-22T20:36:03.655 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.655 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:36:03.655 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691873 2026-02-22T20:36:03.655 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691873 2026-02-22T20:36:03.655 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-459561500675 4-107374182435 5-455266533379 6-150323855394 7-171798691873' 2026-02-22T20:36:03.655 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.655 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:36:03.655 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528352 2026-02-22T20:36:03.656 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528352 2026-02-22T20:36:03.656 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-459561500675 4-107374182435 5-455266533379 6-150323855394 7-171798691873 8-193273528352' 2026-02-22T20:36:03.656 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.656 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:36:03.656 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860427 2026-02-22T20:36:03.656 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860427 2026-02-22T20:36:03.659 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836519 1-42949672998 2-64424509478 3-45956150067helpers.sh:2258: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2026-02-22T20:36:03.659 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=107374182436 2026-02-22T20:36:03.659 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 107374182436 2026-02-22T20:36:03.659 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-459561500674 4-107374182436' 2026-02-22T20:36:03.659 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.659 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2026-02-22T20:36:03.659 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533378 2026-02-22T20:36:03.659 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533378 2026-02-22T20:36:03.659 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-459561500674 4-107374182436 5-455266533378' 2026-02-22T20:36:03.659 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.660 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.6 flush_pg_stats 2026-02-22T20:36:03.660 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855395 2026-02-22T20:36:03.660 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855395 2026-02-22T20:36:03.660 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-459561500674 4-107374182436 5-455266533378 6-150323855395' 2026-02-22T20:36:03.660 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.660 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.7 flush_pg_stats 2026-02-22T20:36:03.660 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=171798691874 2026-02-22T20:36:03.660 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 171798691874 2026-02-22T20:36:03.660 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-459561500674 4-107374182436 5-455266533378 6-150323855395 7-171798691874' 2026-02-22T20:36:03.660 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.660 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.8 flush_pg_stats 2026-02-22T20:36:03.661 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528353 2026-02-22T20:36:03.661 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528353 2026-02-22T20:36:03.661 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-459561500674 4-107374182436 5-455266533378 6-150323855395 7-171798691874 8-193273528353' 2026-02-22T20:36:03.661 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:03.661 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.9 flush_pg_stats 2026-02-22T20:36:03.661 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860428 2026-02-22T20:36:03.661 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860428 2026-02-22T20:36:03.661 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-42949672997 2-64424509477 3-459561500674 4-107374182436 5-455266533378 6-150323855395 7-171798691874 8-193273528353 9-412316860428' 2026-02-22T20:36:03.661 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.160 INFO:tasks.workunit.client.0.trial150.stderr:25 4-107374182435 5-455266533379 6-150323855394 7-171798691873 8-193273528352 9-412316860427' 2026-02-22T20:36:05.160 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.160 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836519 2026-02-22T20:36:05.160 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.160 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:36:05.160 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836519 2026-02-22T20:36:05.160 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:05.160 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836519 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: waiting osd.0 seq 21474836519 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836517 -lt 21474836519 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836519 -lt 21474836519 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672998 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:36:05.161 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672998 2026-02-22T20:36:05.162 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:05.162 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672998 2026-02-22T20:36:05.162 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2026-02-22T20:36:05.162 INFO:tasks.workunit.client.0.trial150.stderr:235165: waiting osd.1 seq 42949672998 2026-02-22T20:36:05.162 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:36:05.162 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672998 -lt 42949672998 2026-02-22T20:36:05.162 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.162 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509478 2026-02-22T20:36:05.162 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.162 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:36:05.162 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509478 2026-02-22T20:36:05.162 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:05.163 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509478 2026-02-22T20:36:05.163 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2026-02-22T20:36:05.163 INFO:tasks.workunit.client.0.trial150.stderr:235165: waiting osd.2 seq 64424509478 2026-02-22T20:36:05.300 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-35167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836518 2026-02-22T20:36:05.301 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.301 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:36:05.301 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836518 2026-02-22T20:36:05.301 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:05.301 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836518 2026-02-22T20:36:05.301 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2026-02-22T20:36:05.301 INFO:tasks.workunit.client.0.trial150.stderr:235167: waiting osd.0 seq 21474836518 2026-02-22T20:36:05.301 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:05.302 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836517 -lt 21474836518 2026-02-22T20:36:05.302 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:36:05.302 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:36:05.302 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:05.302 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836519 -lt 21474836518 2026-02-22T20:36:05.302 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.302 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672997 2026-02-22T20:36:05.302 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.302 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:36:05.302 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672997 2026-02-22T20:36:05.303 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:05.303 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672997 2026-02-22T20:36:05.303 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2026-02-22T20:36:05.303 INFO:tasks.workunit.client.0.trial150.stderr:235167: waiting osd.1 seq 42949672997 2026-02-22T20:36:05.303 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:36:05.303 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672998 -lt 42949672997 2026-02-22T20:36:05.303 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.303 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509477 2026-02-22T20:36:05.303 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.303 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:36:05.303 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509477 2026-02-22T20:36:05.304 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:05.304 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509477 2026-02-22T20:36:05.304 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2026-02-22T20:36:05.304 INFO:tasks.workunit.client.0.trial150.stderr:235167: waiting osd.2 seq 64424509477 2026-02-22T20:36:05.304 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:36:05.304 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509478 -lt 64424509477 2026-02-22T20:36:05.786 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-459561500674 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-459561500674 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=459561500674 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 459561500674' 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: waiting osd.3 seq 459561500674 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 459561500675 -lt 459561500674 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182436 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182436 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182436 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182436' 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: waiting osd.4 seq 107374182436 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:36:05.787 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182436 -lt 107374182436 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-455266533378 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-455266533378 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533378 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 455266533378' 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: waiting osd.5 seq 455266533378 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533379 -lt 455266533378 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855395 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.788 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:36:05.813 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clseq 2 2026-02-22T20:36:05.813 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509478 -lt 64424509478 2026-02-22T20:36:05.814 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.814 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-459561500675 2026-02-22T20:36:05.814 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.814 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:36:05.814 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-459561500675 2026-02-22T20:36:05.814 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:05.814 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=459561500675 2026-02-22T20:36:05.814 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 459561500675' 2026-02-22T20:36:05.814 INFO:tasks.workunit.client.0.trial150.stderr:235165: waiting osd.3 seq 459561500675 2026-02-22T20:36:05.814 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:36:05.814 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 459561500675 -lt 459561500675 2026-02-22T20:36:05.814 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 4-107374182435 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=4 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 4-107374182435 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=107374182435 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.4 seq 107374182435' 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: waiting osd.4 seq 107374182435 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 107374182436 -lt 107374182435 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 5-455266533379 2026-02-22T20:36:05.815 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:05.816 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=5 2026-02-22T20:36:05.816 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 5-455266533379 2026-02-22T20:36:05.816 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:05.816 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533379 2026-02-22T20:36:05.816 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.5 seq 455266533379' 2026-02-22T20:36:05.816 INFO:tasks.workunit.client.0.trial150.stderr:235165: waiting osd.5 seq 455266533379 2026-02-22T20:36:05.816 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-22T20:36:05.816 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533379 -lt 455266533379 2026-02-22T20:36:05.816 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:05.816 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 6-150323855394 2026-02-22T20:36:06.287 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.clientone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855395 2026-02-22T20:36:06.287 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:06.288 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855395 2026-02-22T20:36:06.288 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855395' 2026-02-22T20:36:06.288 INFO:tasks.workunit.client.0.trial150.stderr:235167: waiting osd.6 seq 150323855395 2026-02-22T20:36:06.288 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:36:06.288 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855395 -lt 150323855395 2026-02-22T20:36:06.288 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:06.288 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691874 2026-02-22T20:36:06.288 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:06.288 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:36:06.288 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691874 2026-02-22T20:36:06.289 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:06.289 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691874 2026-02-22T20:36:06.289 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691874' 2026-02-22T20:36:06.289 INFO:tasks.workunit.client.0.trial150.stderr:235167: waiting osd.7 seq 171798691874 2026-02-22T20:36:06.289 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:36:06.289 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691874 -lt 171798691874 2026-02-22T20:36:06.289 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:06.289 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528353 2026-02-22T20:36:06.289 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:06.289 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:36:06.289 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528353 2026-02-22T20:36:06.290 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:06.290 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528353 2026-02-22T20:36:06.290 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528353' 2026-02-22T20:36:06.290 INFO:tasks.workunit.client.0.trial150.stderr:235167: waiting osd.8 seq 193273528353 2026-02-22T20:36:06.290 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:36:06.290 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528354 -lt 193273528353 2026-02-22T20:36:06.290 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:06.290 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-412316860428 2026-02-22T20:36:06.290 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:06.290 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:36:06.290 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-412316860428 2026-02-22T20:36:06.290 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:06.291 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860428 2026-02-22T20:36:06.310 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: .0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:06.310 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=6 2026-02-22T20:36:06.310 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 6-150323855394 2026-02-22T20:36:06.311 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:06.311 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855394 2026-02-22T20:36:06.311 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.6 seq 150323855394' 2026-02-22T20:36:06.311 INFO:tasks.workunit.client.0.trial150.stderr:235165: waiting osd.6 seq 150323855394 2026-02-22T20:36:06.311 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-22T20:36:06.311 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855395 -lt 150323855394 2026-02-22T20:36:06.311 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:06.311 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 7-171798691873 2026-02-22T20:36:06.311 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:06.311 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=7 2026-02-22T20:36:06.311 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 7-171798691873 2026-02-22T20:36:06.312 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:06.312 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=171798691873 2026-02-22T20:36:06.312 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.7 seq 171798691873' 2026-02-22T20:36:06.312 INFO:tasks.workunit.client.0.trial150.stderr:235165: waiting osd.7 seq 171798691873 2026-02-22T20:36:06.312 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-22T20:36:06.312 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 171798691874 -lt 171798691873 2026-02-22T20:36:06.312 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:06.312 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 8-193273528352 2026-02-22T20:36:06.312 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:06.312 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=8 2026-02-22T20:36:06.312 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 8-193273528352 2026-02-22T20:36:06.313 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:06.313 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528352 2026-02-22T20:36:06.313 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.8 seq 193273528352' 2026-02-22T20:36:06.313 INFO:tasks.workunit.client.0.trial150.stderr:235165: waiting osd.8 seq 193273528352 2026-02-22T20:36:06.313 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-22T20:36:06.313 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528354 -lt 193273528352 2026-02-22T20:36:06.313 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:06.313 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 9-412316860427 2026-02-22T20:36:06.313 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:06.313 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=9 2026-02-22T20:36:06.314 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 9-412316860427 2026-02-22T20:36:06.314 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:07.127 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860427 2026-02-22T20:36:07.127 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.9 seq 412316860427' 2026-02-22T20:36:07.128 INFO:tasks.workunit.client.0.trial150.stderr:235165: waiting osd.9 seq 412316860427 2026-02-22T20:36:07.128 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:36:07.128 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860428 -lt 412316860427 2026-02-22T20:36:07.128 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:36:07.128 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:07.128 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:07.128 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:36:07.128 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:36:07.128 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:36:07.128 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:36:07.128 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:36:07.129 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:36:07.129 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:36:07.129 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:36:07.129 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:36:07.129 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:36:07.129 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:07.129 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:07.129 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:36:07.129 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-22T20:36:07.130 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:36:07.130 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-22T20:36:07.130 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:36:07.130 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:36:07.130 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:36:07.130 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:36:07.130 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:36:07.130 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:36:07.130 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:36:07.130 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:36:07.130 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:36:07.256 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/cloneecho 'waiting osd.9 seq 412316860428' 2026-02-22T20:36:07.256 INFO:tasks.workunit.client.0.trial150.stderr:235167: waiting osd.9 seq 412316860428 2026-02-22T20:36:07.256 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-22T20:36:07.256 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860428 -lt 412316860428 2026-02-22T20:36:07.256 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:36:07.256 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:07.257 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:07.257 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:36:07.257 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:36:07.257 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:36:07.257 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:36:07.257 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:36:07.257 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:36:07.257 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:36:07.257 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:36:07.257 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:36:07.257 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:36:07.257 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' -1 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=4 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:36:07.258 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:36:07.259 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:36:07.259 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:36:08.046 INFO:tasks.workunit.client.0.trial150.stderr:235167: /hom.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:36:08.046 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:36:08.046 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:36:08.046 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:08.046 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:08.046 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:36:08.046 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-22T20:36:08.047 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:36:08.047 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:36:08.047 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:36:08.047 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:36:08.047 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:36:08.047 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:36:08.047 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:36:08.047 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:36:08.047 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:36:08.047 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:36:08.047 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:36:08.047 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:36:08.048 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:36:08.048 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:36:08.048 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:36:08.048 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:36:08.048 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:36:08.048 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:36:08.048 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:36:08.048 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:36:08.048 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:36:08.048 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:36:08.048 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:08.197 INFO:tasks.workunit.client.0.trial150.stderr:235165: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:36:08.197 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:36:08.198 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:08.198 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:08.198 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 5 2026-02-22T20:36:08.198 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 4 '!=' 4 2026-02-22T20:36:08.198 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:36:08.198 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:36:08.198 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:36:08.198 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:36:08.198 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:36:08.198 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:36:08.199 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:36:08.199 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:36:08.199 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:36:08.199 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:36:08.199 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:36:08.199 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:36:08.199 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:36:08.199 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:36:08.199 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:36:08.199 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:36:08.199 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:36:08.199 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:36:08.200 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:36:08.200 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:36:08.200 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:36:08.200 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:36:08.200 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:08.200 INFO:tasks.workunit.client.0.trial150.stderr:235167: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:08.200 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:36:08.200 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:36:08.200 INFO:tasks.workunit.client.0.trial150.stderr:235167: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:36:08.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:36:08.239 INFO:tasks.workunit.client.0.trial150.stderr:e/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:08.239 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:36:08.239 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:36:08.239 INFO:tasks.workunit.client.0.trial150.stderr:235165: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:36:08.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:36:08.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:36:08.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 235163 2026-02-22T20:36:08.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:36:08.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:36:08.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:36:08.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:272: corrupt_and_repair_two: return_code=0 2026-02-22T20:36:08.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:273: corrupt_and_repair_two: '[' 0 -ne 0 ']' 2026-02-22T20:36:08.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:275: corrupt_and_repair_two: rados --pool ecpool get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:36:08.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:276: corrupt_and_repair_two: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:36:08.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:36:08.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:36:08.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:36:08.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:36:08.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:36:08.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:36:08.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:36:08.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:36:08.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:36:08.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:36:08.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:36:08.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:36:08.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:36:08.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:36:08.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:36:08.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:36:08.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:36:08.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:36:08.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:36:08.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:36:08.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:36:08.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:36:08.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:36:08.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:36:08.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:08.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:08.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:36:08.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:36:08.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:36:08.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:36:08.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:36:08.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:36:08.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:36:08.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:36:08.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:36:08.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:36:08.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:36:08.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:36:08.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:36:08.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:36:08.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:36:08.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:36:08.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:36:08.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:36:08.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:36:08.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:36:08.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:36:08.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:36:08.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:36:08.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:36:08.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:36:08.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:36:08.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:36:08.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:36:08.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:36:08.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:36:08.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:08.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:08.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:36:08.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:36:08.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:36:08.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:36:08.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:36:08.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:08.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:08.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:36:08.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:36:08.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:36:08.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:36:08.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:36:08.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_and_repair_replicated td/osd-scrub-repair 2026-02-22T20:36:08.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:94: TEST_corrupt_and_repair_replicated: local dir=td/osd-scrub-repair 2026-02-22T20:36:08.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:95: TEST_corrupt_and_repair_replicated: local poolname=rbd 2026-02-22T20:36:08.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:97: TEST_corrupt_and_repair_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-22T20:36:08.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:36:08.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:36:08.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:36:08.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:36:08.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:36:08.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-22T20:36:08.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:36:08.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:08.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:08.665 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:08.666 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:08.666 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:08.666 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:08.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-22T20:36:08.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:36:08.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:36:08.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:36:08.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:36:08.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:36:08.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:36:08.698 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:36:08.698 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:36:08.698 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:36:08.698 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:36:08.698 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:08.699 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:08.699 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:36:08.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:36:08.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:36:08.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:36:08.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:36:08.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:36:08.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:36:08.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:36:08.755 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:36:08.755 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:36:08.755 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:36:08.756 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:36:08.756 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:08.756 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:08.756 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:36:08.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:36:08.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:36:08.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:98: TEST_corrupt_and_repair_replicated: run_mgr td/osd-scrub-repair x 2026-02-22T20:36:08.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:36:08.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:36:08.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:36:08.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:36:08.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:36:08.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:36:08.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:36:08.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:08.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:08.909 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:08.909 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:08.909 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:08.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:08.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:36:08.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:36:08.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:99: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 0 2026-02-22T20:36:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:36:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:36:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:36:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:36:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:36:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:36:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:36:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:36:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:36:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:36:08.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:36:08.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:36:08.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:36:08.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:36:08.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:36:08.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:08.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:08.933 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:08.933 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:08.933 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:08.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:36:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:36:08.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:36:08.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e3da3ddc-6c25-4f10-8c83-23ff1f34c632 2026-02-22T20:36:08.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e3da3ddc-6c25-4f10-8c83-23ff1f34c632' 2026-02-22T20:36:08.936 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 e3da3ddc-6c25-4f10-8c83-23ff1f34c632 2026-02-22T20:36:08.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:36:08.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC4aJtp91O1NxAA41U5LdAd4K0+7gpIhB/TVQ== 2026-02-22T20:36:08.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC4aJtp91O1NxAA41U5LdAd4K0+7gpIhB/TVQ=="}' 2026-02-22T20:36:08.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e3da3ddc-6c25-4f10-8c83-23ff1f34c632 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:36:09.036 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:36:09.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:36:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4aJtp91O1NxAA41U5LdAd4K0+7gpIhB/TVQ== --osd-uuid e3da3ddc-6c25-4f10-8c83-23ff1f34c632 2026-02-22T20:36:09.062 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:09.050+0000 7fd454e17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:09.063 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:09.050+0000 7fd454e17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:09.064 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:09.050+0000 7fd454e17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:09.065 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:09.054+0000 7fd454e17a40 -1 bdev(0x55999cb5ac00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:36:09.065 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:09.054+0000 7fd454e17a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:36:09.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:36:09.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:36:09.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:36:09.833 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:36:09.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:36:10.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:36:10.041 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:36:10.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:36:10.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:36:10.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:36:10.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:36:10.067 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:10.054+0000 7f577507ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:10.069 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:10.058+0000 7f577507ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:10.070 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:10.058+0000 7f577507ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:10.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:36:10.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:36:10.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:36:10.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:36:10.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:36:10.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:10.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:36:10.203 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:36:10.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:10.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:36:10.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:10.431 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:10.418+0000 7f577507ca40 -1 Falling back to public interface 2026-02-22T20:36:11.043 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:11.030+0000 7f577507ca40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:36:11.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:11.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:11.367 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:36:11.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:36:11.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:11.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:36:11.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:12.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:12.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:12.528 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:36:12.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:36:12.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:12.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:36:12.687 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4000309410,v1:127.0.0.1:6803/4000309410] [v2:127.0.0.1:6804/4000309410,v1:127.0.0.1:6805/4000309410] exists,up e3da3ddc-6c25-4f10-8c83-23ff1f34c632 2026-02-22T20:36:12.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:36:12.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:36:12.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:36:12.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:100: TEST_corrupt_and_repair_replicated: run_osd td/osd-scrub-repair 1 2026-02-22T20:36:12.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:36:12.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:36:12.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:36:12.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:36:12.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:36:12.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:36:12.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:36:12.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:36:12.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:36:12.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:36:12.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:36:12.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:36:12.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:36:12.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:36:12.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:36:12.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:12.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:12.692 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:12.692 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:12.692 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:12.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:12.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:12.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:36:12.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:36:12.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:36:12.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:36:12.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:36:12.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:36:12.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:36:12.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:36:12.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:36:12.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:36:12.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:36:12.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:36:12.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:36:12.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3a2022dd-29b7-4d4b-8f94-1c191f339f5c 2026-02-22T20:36:12.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3a2022dd-29b7-4d4b-8f94-1c191f339f5c' 2026-02-22T20:36:12.695 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 3a2022dd-29b7-4d4b-8f94-1c191f339f5c 2026-02-22T20:36:12.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:36:12.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC8aJtpHLNaKRAAJgVpJBVMOAtCw3kOiMEGbg== 2026-02-22T20:36:12.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC8aJtpHLNaKRAAJgVpJBVMOAtCw3kOiMEGbg=="}' 2026-02-22T20:36:12.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3a2022dd-29b7-4d4b-8f94-1c191f339f5c -i td/osd-scrub-repair/1/new.json 2026-02-22T20:36:12.866 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:36:12.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:36:12.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC8aJtpHLNaKRAAJgVpJBVMOAtCw3kOiMEGbg== --osd-uuid 3a2022dd-29b7-4d4b-8f94-1c191f339f5c 2026-02-22T20:36:12.896 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:12.882+0000 7f8019502a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:12.897 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:12.886+0000 7f8019502a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:12.898 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:12.886+0000 7f8019502a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:12.899 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:12.886+0000 7f8019502a40 -1 bdev(0x563a22b1cc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:36:12.899 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:12.886+0000 7f8019502a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:36:13.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:36:13.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:36:13.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:36:13.544 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:36:13.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:36:13.738 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:36:13.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:36:13.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:36:13.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:36:13.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:36:13.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:36:13.754 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:13.742+0000 7fb848689a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:13.756 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:13.742+0000 7fb848689a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:13.757 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:13.746+0000 7fb848689a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:13.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:36:13.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:36:13.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:36:13.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:36:13.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:36:13.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:13.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:36:13.893 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:36:13.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:13.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:36:14.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:14.171 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:14.158+0000 7fb848689a40 -1 Falling back to public interface 2026-02-22T20:36:14.787 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:14.774+0000 7fb848689a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:36:15.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:15.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:15.052 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:36:15.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:36:15.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:15.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:36:15.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:16.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:16.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:16.218 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:36:16.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:36:16.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:16.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:36:16.379 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2778713056,v1:127.0.0.1:6811/2778713056] [v2:127.0.0.1:6812/2778713056,v1:127.0.0.1:6813/2778713056] exists,up 3a2022dd-29b7-4d4b-8f94-1c191f339f5c 2026-02-22T20:36:16.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:36:16.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:36:16.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:36:16.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:101: TEST_corrupt_and_repair_replicated: create_rbd_pool 2026-02-22T20:36:16.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:36:16.527 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:36:16.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:36:16.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:36:16.754 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:36:16.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:36:17.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:36:18.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:102: TEST_corrupt_and_repair_replicated: wait_for_clean 2026-02-22T20:36:18.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:36:18.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:36:18.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:36:18.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:36:18.051 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:36:18.051 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:36:18.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:36:18.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:36:18.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:36:18.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:36:18.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:36:18.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:36:18.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:36:18.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:36:18.160 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:36:18.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:36:18.315 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:36:18.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:36:18.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:18.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:36:18.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:36:18.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:36:18.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:36:18.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:18.388 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:36:18.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-22T20:36:18.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-22T20:36:18.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-22T20:36:18.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:18.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:36:18.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:18.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:36:18.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:36:18.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:18.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:36:18.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:36:18.465 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:36:18.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:18.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:36:18.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:36:19.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:36:19.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:19.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T20:36:19.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:19.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-22T20:36:19.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:19.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:36:19.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-22T20:36:19.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:19.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-22T20:36:19.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-22T20:36:19.797 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705666 2026-02-22T20:36:19.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:36:19.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-22T20:36:19.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:36:19.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:19.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:20.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:36:20.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:36:20.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:36:20.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:36:20.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:36:20.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:36:20.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:36:20.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:36:20.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:36:20.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:36:20.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:20.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:20.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:36:20.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:36:20.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:36:20.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:104: TEST_corrupt_and_repair_replicated: add_something td/osd-scrub-repair rbd 2026-02-22T20:36:20.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:36:20.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2026-02-22T20:36:20.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-22T20:36:20.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:36:20.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:36:20.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:36:20.697 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:36:20.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:36:20.903 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:36:20.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:36:20.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:36:20.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:36:20.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: get_not_primary rbd SOMETHING 2026-02-22T20:36:20.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-22T20:36:20.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:36:20.943 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2026-02-22T20:36:20.943 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-22T20:36:20.943 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:36:20.943 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-22T20:36:20.944 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:36:21.107 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:36:21.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2026-02-22T20:36:21.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:36:21.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:105: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 0 2026-02-22T20:36:21.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-22T20:36:21.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=rbd 2026-02-22T20:36:21.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=0 2026-02-22T20:36:21.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:36:21.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:36:21.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:36:21.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:36:21.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:36:21.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:36:21.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:36:21.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:36:21.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:36:21.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:36:21.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:36:21.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:36:21.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:36:21.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:36:21.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:36:21.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:36:21.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:36:21.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:36:21.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:36:21.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:36:21.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:36:21.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:36:21.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:36:21.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-22T20:36:21.777 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-22T20:36:21.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:36:21.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:36:21.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:36:21.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:36:21.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:36:21.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:36:21.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:36:21.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:36:21.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:36:21.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:36:21.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:36:21.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:36:21.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:36:21.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:36:21.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:36:21.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:36:21.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:36:21.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:36:21.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:36:21.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:36:21.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:36:21.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:36:21.872 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:36:21.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:36:21.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:36:21.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:36:21.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:36:21.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:36:21.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:36:21.888 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:21.874+0000 7f4c184d9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:21.890 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:21.878+0000 7f4c184d9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:21.890 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:21.878+0000 7f4c184d9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:22.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:36:22.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:36:22.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:36:22.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:36:22.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:36:22.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:22.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:36:22.031 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:36:22.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:22.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:36:22.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:22.227 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:22.214+0000 7f4c184d9a40 -1 Falling back to public interface 2026-02-22T20:36:22.852 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:22.838+0000 7f4c184d9a40 -1 osd.0 18 log_to_monitors true 2026-02-22T20:36:23.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:23.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:23.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:36:23.194 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:36:23.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:23.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:36:23.353 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 22 up_thru 22 down_at 19 last_clean_interval [5,18) [v2:127.0.0.1:6802/1303508195,v1:127.0.0.1:6803/1303508195] [v2:127.0.0.1:6804/1303508195,v1:127.0.0.1:6805/1303508195] exists,up e3da3ddc-6c25-4f10-8c83-23ff1f34c632 2026-02-22T20:36:23.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:36:23.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:36:23.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:36:23.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:36:23.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:36:23.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:36:23.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:36:23.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:36:23.355 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:36:23.355 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:36:23.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:36:23.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:36:23.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:36:23.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:36:23.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:36:23.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:36:23.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:36:23.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:36:23.466 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:36:23.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:36:23.623 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:36:23.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:36:23.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:23.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:36:23.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280514 2026-02-22T20:36:23.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280514 2026-02-22T20:36:23.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514' 2026-02-22T20:36:23.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:23.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:36:23.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-22T20:36:23.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-22T20:36:23.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280514 1-38654705668' 2026-02-22T20:36:23.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:23.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280514 2026-02-22T20:36:23.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:23.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:36:23.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280514 2026-02-22T20:36:23.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:23.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280514 2026-02-22T20:36:23.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280514' 2026-02-22T20:36:23.773 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 94489280514 2026-02-22T20:36:23.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:23.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 94489280514 2026-02-22T20:36:23.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:36:24.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:36:24.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:25.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 94489280514 2026-02-22T20:36:25.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:36:26.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:36:26.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:26.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280514 -lt 94489280514 2026-02-22T20:36:26.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:26.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-22T20:36:26.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:26.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:36:26.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705668 2026-02-22T20:36:26.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:26.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705668 2026-02-22T20:36:26.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-22T20:36:26.268 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 38654705668 2026-02-22T20:36:26.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:36:26.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705668 2026-02-22T20:36:26.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:36:26.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:26.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:26.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:36:26.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:36:26.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:36:26.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:36:26.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:36:26.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:36:26.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:36:26.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:36:26.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:36:26.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:36:26.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:26.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:26.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:36:26.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:36:26.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:36:26.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg rbd SOMETHING 2026-02-22T20:36:26.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-22T20:36:26.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:36:26.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-22T20:36:26.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:36:27.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=1.3 2026-02-22T20:36:27.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 1.3 2026-02-22T20:36:27.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.3 2026-02-22T20:36:27.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.3 2026-02-22T20:36:27.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:36:27.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:27.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:27.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:36:27.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:36:16.744915+0000 2026-02-22T20:36:27.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.3 2026-02-22T20:36:27.460 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.3 on osd.1 to repair 2026-02-22T20:36:27.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.3 2026-02-22T20:36:16.744915+0000 2026-02-22T20:36:27.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.3 2026-02-22T20:36:27.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:36:16.744915+0000 2026-02-22T20:36:27.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:36:27.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:36:27.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:36:27.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:36:27.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:36:27.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:27.472 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:27.472 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:36:27.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:36:16.744915+0000 '>' 2026-02-22T20:36:16.744915+0000 2026-02-22T20:36:27.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:36:28.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:36:28.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:36:28.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:36:28.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:36:28.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:28.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:28.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:36:28.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:36:16.744915+0000 '>' 2026-02-22T20:36:16.744915+0000 2026-02-22T20:36:28.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:36:29.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:36:29.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:36:29.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:36:29.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:36:29.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:29.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:29.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:36:29.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:36:16.744915+0000 '>' 2026-02-22T20:36:16.744915+0000 2026-02-22T20:36:29.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:36:30.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:36:30.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:36:30.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:36:30.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:36:30.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:30.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:30.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:36:31.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:36:16.744915+0000 '>' 2026-02-22T20:36:16.744915+0000 2026-02-22T20:36:31.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:36:32.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:36:32.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:36:32.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:36:32.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:36:32.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:32.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:32.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:36:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:36:27.785746+0000 '>' 2026-02-22T20:36:16.744915+0000 2026-02-22T20:36:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:36:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:36:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:36:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:36:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:36:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:36:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:36:32.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:36:32.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:36:32.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:36:32.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:36:32.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:36:32.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:36:32.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:36:32.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:36:32.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:36:32.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:36:32.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:36:32.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:36:32.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:36:32.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:36:32.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:36:32.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:36:32.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:36:32.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-22T20:36:32.660 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:36:32.661 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:36:32.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:36:32.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:36:32.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:36:32.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:36:32.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:36:32.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:36:32.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:36:32.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:36:32.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:36:32.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:36:32.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:36:32.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:36:32.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:36:32.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:36:32.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:36:32.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:36:32.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:36:32.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:36:32.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:36:32.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:36:32.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:36:32.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:36:32.759 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:36:32.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:36:32.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:36:32.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:36:32.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:36:32.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:36:32.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:36:32.774 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:32.762+0000 7ffb93de4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:32.776 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:32.762+0000 7ffb93de4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:32.777 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:32.762+0000 7ffb93de4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:32.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:36:32.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:36:32.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:36:32.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:36:32.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:36:32.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:32.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:36:32.916 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:36:32.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:32.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:36:33.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:33.159 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:33.146+0000 7ffb93de4a40 -1 Falling back to public interface 2026-02-22T20:36:33.767 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:33.754+0000 7ffb93de4a40 -1 osd.1 27 log_to_monitors true 2026-02-22T20:36:34.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:34.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:34.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:36:34.080 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:36:34.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:34.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:36:34.236 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 27 last_clean_interval [9,27) [v2:127.0.0.1:6810/2889845849,v1:127.0.0.1:6811/2889845849] [v2:127.0.0.1:6812/2889845849,v1:127.0.0.1:6813/2889845849] exists,up 3a2022dd-29b7-4d4b-8f94-1c191f339f5c 2026-02-22T20:36:34.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:36:34.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:36:34.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:36:34.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:36:34.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:36:34.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:36:34.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:36:34.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:36:34.238 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:36:34.238 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:36:34.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:36:34.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:36:34.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:36:34.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:36:34.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:36:34.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:36:34.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:36:34.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:36:34.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:36:34.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:36:34.508 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:36:34.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:36:34.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:34.508 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:36:34.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280517 2026-02-22T20:36:34.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280517 2026-02-22T20:36:34.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280517' 2026-02-22T20:36:34.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:34.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:36:34.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-22T20:36:34.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-22T20:36:34.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280517 1-128849018882' 2026-02-22T20:36:34.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:34.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280517 2026-02-22T20:36:34.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:34.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:36:34.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280517 2026-02-22T20:36:34.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:34.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280517 2026-02-22T20:36:34.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280517' 2026-02-22T20:36:34.657 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 94489280517 2026-02-22T20:36:34.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:34.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280516 -lt 94489280517 2026-02-22T20:36:34.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:36:35.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:36:35.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:35.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280517 -lt 94489280517 2026-02-22T20:36:35.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:35.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-128849018882 2026-02-22T20:36:35.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:35.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:36:35.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-128849018882 2026-02-22T20:36:35.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:35.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-22T20:36:35.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 128849018882' 2026-02-22T20:36:35.985 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 128849018882 2026-02-22T20:36:35.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:36:36.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018882 -lt 128849018882 2026-02-22T20:36:36.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:36:36.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:36.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:36.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:36:36.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:36:36.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:36:36.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:36:36.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:36:36.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:36:36.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:36:36.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:36:36.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:36:36.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:36:36.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:36.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:36.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:36:36.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:36:36.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:36:36.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:36:36.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:36:36.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: get_primary rbd SOMETHING 2026-02-22T20:36:36.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-22T20:36:36.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:36:36.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-22T20:36:36.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:107: TEST_corrupt_and_repair_replicated: corrupt_and_repair_one td/osd-scrub-repair rbd 1 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:286: corrupt_and_repair_one: local dir=td/osd-scrub-repair 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:287: corrupt_and_repair_one: local poolname=rbd 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:288: corrupt_and_repair_one: local osd=1 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:293: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:36:36.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:36:36.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:36:37.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:36:37.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING remove 2026-02-22T20:36:37.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:36:37.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:36:37.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:36:37.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:36:37.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:36:37.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING remove 2026-02-22T20:36:37.360 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:eb822e21:::SOMETHING:head# 2026-02-22T20:36:37.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:36:37.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:36:37.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:36:37.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:36:37.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:36:37.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:36:37.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:36:37.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:36:37.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:36:37.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:36:37.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:36:37.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:36:37.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:36:37.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:36:37.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:36:37.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:36:37.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:37.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:37.463 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:37.463 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:37.463 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:37.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:37.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:37.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:36:37.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:36:37.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:36:37.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:36:37.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:36:37.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:36:37.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:36:37.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:36:37.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:36:37.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:36:37.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:36:37.466 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:36:37.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:36:37.466 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:36:37.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:36:37.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:36:37.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:36:37.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:36:37.483 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:37.470+0000 7ff230aaba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:37.484 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:37.470+0000 7ff230aaba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:37.485 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:37.474+0000 7ff230aaba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:37.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:36:37.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:36:37.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:36:37.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:36:37.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:36:37.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:37.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:36:37.618 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:36:37.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:37.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:36:37.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:37.911 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:37.898+0000 7ff230aaba40 -1 Falling back to public interface 2026-02-22T20:36:38.511 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:38.498+0000 7ff230aaba40 -1 osd.1 31 log_to_monitors true 2026-02-22T20:36:38.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:38.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:38.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:36:38.781 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:36:38.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:38.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:36:38.942 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [30,31) [v2:127.0.0.1:6810/3141096208,v1:127.0.0.1:6811/3141096208] [v2:127.0.0.1:6812/3141096208,v1:127.0.0.1:6813/3141096208] exists,up 3a2022dd-29b7-4d4b-8f94-1c191f339f5c 2026-02-22T20:36:38.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:36:38.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:36:38.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:36:38.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:36:38.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:36:38.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:36:38.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:36:38.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:36:38.944 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:36:38.944 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:36:38.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:36:38.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:36:38.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:36:39.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:36:39.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:36:39.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:36:39.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:36:39.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:36:39.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:36:39.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:36:39.209 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:36:39.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:36:39.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:39.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:36:39.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280519 2026-02-22T20:36:39.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280519 2026-02-22T20:36:39.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280519' 2026-02-22T20:36:39.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:39.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:36:39.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855362 2026-02-22T20:36:39.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855362 2026-02-22T20:36:39.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280519 1-150323855362' 2026-02-22T20:36:39.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:39.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280519 2026-02-22T20:36:39.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:39.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:36:39.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280519 2026-02-22T20:36:39.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:39.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280519 2026-02-22T20:36:39.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280519' 2026-02-22T20:36:39.360 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 94489280519 2026-02-22T20:36:39.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:39.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280519 -lt 94489280519 2026-02-22T20:36:39.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:39.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-150323855362 2026-02-22T20:36:39.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:39.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:36:39.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-150323855362 2026-02-22T20:36:39.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:39.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855362 2026-02-22T20:36:39.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2026-02-22T20:36:39.533 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 150323855362 2026-02-22T20:36:39.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:36:39.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855362 -lt 150323855362 2026-02-22T20:36:39.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:36:39.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:39.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:39.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:36:39.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:36:39.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:36:39.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:36:39.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:36:39.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:36:39.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:36:39.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:36:40.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:36:40.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:36:40.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:40.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:40.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:36:40.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:36:40.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:36:40.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: get_pg rbd SOMETHING 2026-02-22T20:36:40.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-22T20:36:40.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:36:40.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-22T20:36:40.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:36:40.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:297: corrupt_and_repair_one: local pg=1.3 2026-02-22T20:36:40.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:298: corrupt_and_repair_one: repair 1.3 2026-02-22T20:36:40.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.3 2026-02-22T20:36:40.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.3 2026-02-22T20:36:40.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:36:40.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:40.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:40.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:36:40.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:36:27.785746+0000 2026-02-22T20:36:40.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.3 2026-02-22T20:36:40.721 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.3 on osd.1 to repair 2026-02-22T20:36:40.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.3 2026-02-22T20:36:27.785746+0000 2026-02-22T20:36:40.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.3 2026-02-22T20:36:40.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:36:27.785746+0000 2026-02-22T20:36:40.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:36:40.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:36:40.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:36:40.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:36:40.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:36:40.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:40.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:40.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:36:40.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:36:27.785746+0000 '>' 2026-02-22T20:36:27.785746+0000 2026-02-22T20:36:40.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:36:41.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:36:41.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:36:41.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:36:41.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:36:41.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:41.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:41.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:36:42.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:36:27.785746+0000 '>' 2026-02-22T20:36:27.785746+0000 2026-02-22T20:36:42.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:36:43.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:36:43.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:36:43.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:36:43.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:36:43.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:43.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:43.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:36:43.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:36:27.785746+0000 '>' 2026-02-22T20:36:27.785746+0000 2026-02-22T20:36:43.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:36:44.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:36:44.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:36:44.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:36:44.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:36:44.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:44.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:44.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:36:44.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:36:27.785746+0000 '>' 2026-02-22T20:36:27.785746+0000 2026-02-22T20:36:44.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:36:45.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:36:45.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:36:45.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:36:45.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:36:45.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:36:45.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:36:45.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:36:45.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:36:41.563854+0000 '>' 2026-02-22T20:36:27.785746+0000 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:302: corrupt_and_repair_one: objectstore_tool td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:36:45.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:36:45.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:36:45.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 SOMETHING list-attrs 2026-02-22T20:36:45.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:36:45.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:36:45.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:36:45.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:36:45.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:36:45.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 SOMETHING list-attrs 2026-02-22T20:36:45.927 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::deep_scrub_1.3:head#, (61) No data available 2026-02-22T20:36:45.928 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.3_head,#-3:c0000000:::scrub_1.3:head#, (61) No data available 2026-02-22T20:36:45.928 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:36:45.928 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:36:46.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:36:46.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:36:46.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:36:46.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:36:46.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:36:46.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:36:46.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:36:46.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:36:46.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:36:46.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:36:46.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:36:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:36:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:36:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:36:46.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:36:46.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:36:46.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:46.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:46.027 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:46.027 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:46.027 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:46.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:46.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:46.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:36:46.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:36:46.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:36:46.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:36:46.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:36:46.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:36:46.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:36:46.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:36:46.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:36:46.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:36:46.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:36:46.029 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:36:46.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:36:46.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:36:46.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:36:46.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:36:46.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:36:46.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:36:46.046 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:46.033+0000 7f9e8173ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:46.047 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:46.033+0000 7f9e8173ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:46.048 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:46.033+0000 7f9e8173ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:46.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:36:46.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:36:46.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:36:46.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:36:46.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:36:46.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:46.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:36:46.188 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:36:46.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:46.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:36:46.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:46.415 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:46.401+0000 7f9e8173ba40 -1 Falling back to public interface 2026-02-22T20:36:47.040 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:47.025+0000 7f9e8173ba40 -1 osd.1 37 log_to_monitors true 2026-02-22T20:36:47.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:47.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:47.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:36:47.349 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:36:47.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:47.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:36:47.504 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 38 last_clean_interval [35,37) [v2:127.0.0.1:6810/3763533157,v1:127.0.0.1:6811/3763533157] [v2:127.0.0.1:6812/3763533157,v1:127.0.0.1:6813/3763533157] exists,up 3a2022dd-29b7-4d4b-8f94-1c191f339f5c 2026-02-22T20:36:47.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:36:47.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:36:47.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:36:47.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:36:47.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:36:47.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:36:47.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:36:47.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:36:47.506 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:36:47.506 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:36:47.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:36:47.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:36:47.507 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:36:47.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:36:47.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:36:47.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:36:47.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:36:47.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:36:47.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:36:47.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:36:47.767 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:36:47.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:36:47.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:47.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:36:47.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280521 2026-02-22T20:36:47.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280521 2026-02-22T20:36:47.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280521' 2026-02-22T20:36:47.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:36:47.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:36:47.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659138 2026-02-22T20:36:47.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659138 2026-02-22T20:36:47.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-94489280521 1-176093659138' 2026-02-22T20:36:47.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:47.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-94489280521 2026-02-22T20:36:47.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:47.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:36:47.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-94489280521 2026-02-22T20:36:47.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:47.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280521 2026-02-22T20:36:47.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 94489280521' 2026-02-22T20:36:47.919 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 94489280521 2026-02-22T20:36:47.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:48.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280520 -lt 94489280521 2026-02-22T20:36:48.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:36:49.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:36:49.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:49.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280520 -lt 94489280521 2026-02-22T20:36:49.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:36:50.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:36:50.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:36:50.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280522 -lt 94489280521 2026-02-22T20:36:50.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:36:50.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-176093659138 2026-02-22T20:36:50.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:36:50.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:36:50.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-176093659138 2026-02-22T20:36:50.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:36:50.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659138 2026-02-22T20:36:50.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2026-02-22T20:36:50.403 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 176093659138 2026-02-22T20:36:50.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:36:50.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659138 -lt 176093659138 2026-02-22T20:36:50.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:36:50.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:50.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:50.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:36:50.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:36:50.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:36:50.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:36:50.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:36:50.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:36:50.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:36:50.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:36:50.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:36:50.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:36:50.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:36:50.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:36:51.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:36:51.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:36:51.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:36:51.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:303: corrupt_and_repair_one: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:36:51.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:304: corrupt_and_repair_one: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:36:51.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:36:51.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:36:51.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:36:51.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:36:51.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:36:51.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:36:51.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:36:51.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:36:51.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:36:51.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:36:51.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:36:51.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:36:51.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:36:51.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:36:51.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:36:51.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:36:51.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:36:51.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:36:51.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:36:51.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:36:51.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:36:51.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:36:51.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:36:51.254 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:36:51.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:51.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:51.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:36:51.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:36:51.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:36:51.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:36:51.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:36:51.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:36:51.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:36:51.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:36:51.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:36:51.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:36:51.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:36:51.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:36:51.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:36:51.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:36:51.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:36:51.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:36:51.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:36:51.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:36:51.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:36:51.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:36:51.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:36:51.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:36:51.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:36:51.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:36:51.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:36:51.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:36:51.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:36:51.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:36:51.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:36:51.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:36:51.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:51.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:51.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:36:51.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:36:51.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:36:51.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:36:51.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:36:51.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:51.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:51.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:36:51.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:36:51.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:36:51.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:36:51.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:36:51.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_appends td/osd-scrub-repair 2026-02-22T20:36:51.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5912: TEST_corrupt_scrub_erasure_appends: corrupt_scrub_erasure td/osd-scrub-repair false 2026-02-22T20:36:51.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3736: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2026-02-22T20:36:51.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3737: corrupt_scrub_erasure: local allow_overwrites=false 2026-02-22T20:36:51.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3738: corrupt_scrub_erasure: local poolname=ecpool 2026-02-22T20:36:51.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3739: corrupt_scrub_erasure: local total_objs=7 2026-02-22T20:36:51.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3741: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2026-02-22T20:36:51.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:36:51.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:36:51.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:36:51.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:36:51.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:36:51.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:36:51.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:36:51.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:51.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:51.297 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:51.297 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:51.297 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:51.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:51.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:36:51.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:36:51.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:36:51.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:36:51.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:36:51.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:36:51.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:36:51.325 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:36:51.325 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:36:51.325 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:36:51.325 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:36:51.326 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:51.326 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:51.326 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:36:51.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:36:51.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:36:51.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:36:51.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:36:51.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:36:51.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:36:51.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:36:51.382 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:36:51.383 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:36:51.383 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:36:51.383 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:36:51.383 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:51.383 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:51.383 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:36:51.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:36:51.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:36:51.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3742: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2026-02-22T20:36:51.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:36:51.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:36:51.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:36:51.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:36:51.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:36:51.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:36:51.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:36:51.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:51.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:51.535 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:51.535 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:51.535 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:51.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:51.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:36:51.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:36:51.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: seq 0 2 2026-02-22T20:36:51.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-22T20:36:51.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2026-02-22T20:36:51.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:36:51.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:36:51.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:36:51.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:36:51.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:36:51.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:36:51.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:36:51.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:36:51.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:36:51.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:36:51.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:36:51.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:36:51.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:36:51.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:36:51.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:36:51.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:51.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:51.561 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:51.562 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:51.562 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:51.562 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:51.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:51.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:36:51.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:36:51.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:36:51.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:36:51.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:36:51.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:36:51.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:36:51.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:36:51.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:36:51.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:36:51.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:36:51.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:36:51.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:36:51.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=66150e60-7a54-49c2-a824-b3bc1b80d828 2026-02-22T20:36:51.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 66150e60-7a54-49c2-a824-b3bc1b80d828' 2026-02-22T20:36:51.565 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 66150e60-7a54-49c2-a824-b3bc1b80d828 2026-02-22T20:36:51.566 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:36:51.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDjaJtpekuvIRAASDC75/ibTrfO7vmDzWkXGQ== 2026-02-22T20:36:51.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDjaJtpekuvIRAASDC75/ibTrfO7vmDzWkXGQ=="}' 2026-02-22T20:36:51.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 66150e60-7a54-49c2-a824-b3bc1b80d828 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:36:51.664 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:36:51.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:36:51.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDjaJtpekuvIRAASDC75/ibTrfO7vmDzWkXGQ== --osd-uuid 66150e60-7a54-49c2-a824-b3bc1b80d828 2026-02-22T20:36:51.688 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:51.673+0000 7f72375f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:51.690 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:51.677+0000 7f72375f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:51.691 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:51.677+0000 7f72375f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:51.691 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:51.677+0000 7f72375f9a40 -1 bdev(0x55a8a31bec00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:36:51.691 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:51.677+0000 7f72375f9a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:36:52.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:36:52.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:36:52.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:36:52.463 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:36:52.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:36:52.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:36:52.667 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:36:52.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:36:52.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:36:52.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:36:52.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:36:52.683 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:52.669+0000 7f8229e97a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:52.685 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:52.669+0000 7f8229e97a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:52.686 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:52.673+0000 7f8229e97a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:52.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:36:52.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:36:52.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:36:52.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:36:52.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:36:52.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:52.837 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:36:52.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:36:52.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:52.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:36:52.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:53.027 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:53.013+0000 7f8229e97a40 -1 Falling back to public interface 2026-02-22T20:36:53.592 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:53.577+0000 7f8229e97a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:36:53.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:53.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:53.990 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:36:53.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:36:53.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:53.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:36:54.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:55.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:55.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:55.151 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:36:55.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:36:55.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:55.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:36:55.311 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2382775672,v1:127.0.0.1:6803/2382775672] [v2:127.0.0.1:6804/2382775672,v1:127.0.0.1:6805/2382775672] exists,up 66150e60-7a54-49c2-a824-b3bc1b80d828 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:55.312 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:55.313 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:55.313 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:55.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:55.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:55.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:36:55.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:36:55.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:36:55.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:36:55.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:36:55.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:36:55.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:36:55.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:36:55.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:36:55.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:36:55.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:36:55.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:36:55.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:36:55.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ddce7ffd-1827-4a8c-a1b3-20246bd6a9a4 2026-02-22T20:36:55.316 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 ddce7ffd-1827-4a8c-a1b3-20246bd6a9a4 2026-02-22T20:36:55.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ddce7ffd-1827-4a8c-a1b3-20246bd6a9a4' 2026-02-22T20:36:55.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:36:55.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDnaJtprTv4EhAAzIZtHrTW966q5Ei+tXfAew== 2026-02-22T20:36:55.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDnaJtprTv4EhAAzIZtHrTW966q5Ei+tXfAew=="}' 2026-02-22T20:36:55.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ddce7ffd-1827-4a8c-a1b3-20246bd6a9a4 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:36:55.486 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:36:55.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:36:55.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnaJtprTv4EhAAzIZtHrTW966q5Ei+tXfAew== --osd-uuid ddce7ffd-1827-4a8c-a1b3-20246bd6a9a4 2026-02-22T20:36:55.514 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:55.501+0000 7f4962f8ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:55.515 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:55.501+0000 7f4962f8ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:55.516 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:55.501+0000 7f4962f8ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:55.517 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:55.501+0000 7f4962f8ea40 -1 bdev(0x56340ae20c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:36:55.517 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:55.505+0000 7f4962f8ea40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:36:56.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:36:56.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:36:56.137 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:36:56.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:36:56.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:36:56.345 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:36:56.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:36:56.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:36:56.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:36:56.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:36:56.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:36:56.361 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:56.349+0000 7f237cfd6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:56.363 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:56.349+0000 7f237cfd6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:56.364 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:56.349+0000 7f237cfd6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:56.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:36:56.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:36:56.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:36:56.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:36:56.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:36:56.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:56.502 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:36:56.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:36:56.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:56.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:36:56.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:56.771 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:56.757+0000 7f237cfd6a40 -1 Falling back to public interface 2026-02-22T20:36:57.363 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:57.349+0000 7f237cfd6a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:36:57.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:57.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:57.666 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:36:57.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:36:57.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:57.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:36:57.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:36:58.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:36:58.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:36:58.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:36:58.823 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:36:58.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:36:58.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:36:58.982 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4117095289,v1:127.0.0.1:6811/4117095289] [v2:127.0.0.1:6812/4117095289,v1:127.0.0.1:6813/4117095289] exists,up ddce7ffd-1827-4a8c-a1b3-20246bd6a9a4 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:36:58.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:36:58.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:36:58.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:36:58.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:36:58.984 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:36:58.984 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:36:58.984 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:36:58.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:36:58.986 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:36:58.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cee891b2-6c0d-496a-b376-597b5a64bc9c 2026-02-22T20:36:58.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 cee891b2-6c0d-496a-b376-597b5a64bc9c' 2026-02-22T20:36:58.987 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 cee891b2-6c0d-496a-b376-597b5a64bc9c 2026-02-22T20:36:58.987 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:36:58.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDqaJtphCP5OhAAu9SrEHKwEYJBNLJLT1ROdg== 2026-02-22T20:36:59.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDqaJtphCP5OhAAu9SrEHKwEYJBNLJLT1ROdg=="}' 2026-02-22T20:36:59.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cee891b2-6c0d-496a-b376-597b5a64bc9c -i td/osd-scrub-repair/2/new.json 2026-02-22T20:36:59.164 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:36:59.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:36:59.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDqaJtphCP5OhAAu9SrEHKwEYJBNLJLT1ROdg== --osd-uuid cee891b2-6c0d-496a-b376-597b5a64bc9c 2026-02-22T20:36:59.191 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:59.177+0000 7f667e06ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:59.193 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:59.177+0000 7f667e06ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:59.194 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:59.181+0000 7f667e06ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:36:59.194 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:59.181+0000 7f667e06ca40 -1 bdev(0x559c0818cc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:36:59.195 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:36:59.181+0000 7f667e06ca40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:36:59.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:36:59.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:36:59.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:36:59.840 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:36:59.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:37:00.048 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:37:00.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:37:00.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:37:00.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:37:00.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:37:00.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:37:00.072 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:00.057+0000 7fa20c56ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:00.074 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:00.061+0000 7fa20c56ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:00.075 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:00.061+0000 7fa20c56ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:00.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:37:00.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:37:00.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:37:00.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:37:00.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:37:00.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:00.207 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:37:00.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:37:00.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:00.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:37:00.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:37:00.427 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:00.413+0000 7fa20c56ea40 -1 Falling back to public interface 2026-02-22T20:37:01.115 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:01.101+0000 7fa20c56ea40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:37:01.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:37:01.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:01.364 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:37:01.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:37:01.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:01.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:37:01.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:37:02.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:37:02.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:02.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:37:02.529 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:37:02.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:02.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:37:02.685 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2566283542,v1:127.0.0.1:6819/2566283542] [v2:127.0.0.1:6820/2566283542,v1:127.0.0.1:6821/2566283542] exists,up cee891b2-6c0d-496a-b376-597b5a64bc9c 2026-02-22T20:37:02.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:37:02.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:37:02.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:37:02.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3746: corrupt_scrub_erasure: create_rbd_pool 2026-02-22T20:37:02.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:37:02.838 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:37:02.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:37:02.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:37:03.059 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:37:03.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:37:04.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:37:04.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3747: corrupt_scrub_erasure: create_pool foo 1 2026-02-22T20:37:04.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-22T20:37:04.591 INFO:tasks.workunit.client.0.trial150.stderr:pool 'foo' created 2026-02-22T20:37:04.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:37:05.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3749: corrupt_scrub_erasure: create_ec_pool ecpool false k=2 m=1 stripe_unit=2K --force 2026-02-22T20:37:05.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-22T20:37:05.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:37:05.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-22T20:37:05.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:37:05.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2026-02-22T20:37:05.813 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:37:05.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-22T20:37:05.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-22T20:37:06.120 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' created 2026-02-22T20:37:06.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:37:07.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-22T20:37:07.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:37:07.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:37:07.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:37:07.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:37:07.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:37:07.134 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:37:07.134 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:37:07.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:37:07.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:37:07.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:37:07.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:37:07.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:37:07.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:37:07.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:37:07.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:37:07.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:37:07.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:37:07.410 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:07.410 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:37:07.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:37:07.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:07.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:37:07.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-22T20:37:07.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-22T20:37:07.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T20:37:07.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:07.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:37:07.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-22T20:37:07.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-22T20:37:07.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-22T20:37:07.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:07.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:37:07.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-22T20:37:07.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-22T20:37:07.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-60129542147' 2026-02-22T20:37:07.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:07.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-22T20:37:07.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:07.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:37:07.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-22T20:37:07.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:07.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-22T20:37:07.634 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836484 2026-02-22T20:37:07.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T20:37:07.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:07.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:37:07.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:08.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:37:08.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:08.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-22T20:37:08.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:08.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-22T20:37:08.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:08.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:37:08.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-22T20:37:08.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:08.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-22T20:37:08.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-22T20:37:08.972 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705667 2026-02-22T20:37:08.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:37:09.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-22T20:37:09.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:09.145 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-22T20:37:09.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:09.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:37:09.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-22T20:37:09.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:09.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-22T20:37:09.149 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542147 2026-02-22T20:37:09.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-22T20:37:09.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:37:09.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542147 -lt 60129542147 2026-02-22T20:37:09.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:37:09.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:09.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:09.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:37:09.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:37:09.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:37:09.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:37:09.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:37:09.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:37:09.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:37:09.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:37:09.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:37:09.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:37:09.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:09.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:09.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:37:09.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:37:09.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:37:09.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:37:09.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3750: corrupt_scrub_erasure: wait_for_clean 2026-02-22T20:37:09.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:37:09.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:37:09.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:37:09.867 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:37:09.868 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:37:09.868 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:37:09.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:37:09.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:37:09.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:37:09.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:37:09.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:37:09.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:37:09.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:37:09.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:37:09.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:37:10.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:37:10.141 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:10.141 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:37:10.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:37:10.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:10.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:37:10.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-22T20:37:10.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-22T20:37:10.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-22T20:37:10.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:10.213 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:37:10.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-22T20:37:10.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-22T20:37:10.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705669' 2026-02-22T20:37:10.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:10.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:37:10.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-22T20:37:10.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-22T20:37:10.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705669 2-60129542148' 2026-02-22T20:37:10.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:10.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-22T20:37:10.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:10.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:37:10.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-22T20:37:10.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:10.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-22T20:37:10.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-22T20:37:10.361 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836486 2026-02-22T20:37:10.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-22T20:37:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:11.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:37:11.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:11.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-22T20:37:11.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:12.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:37:12.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:12.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-22T20:37:12.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:12.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-22T20:37:12.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:12.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:37:12.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-22T20:37:12.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:12.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-22T20:37:12.858 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705669 2026-02-22T20:37:12.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-22T20:37:12.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:37:13.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-22T20:37:13.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:13.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-22T20:37:13.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:13.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:37:13.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-22T20:37:13.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:13.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-22T20:37:13.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-22T20:37:13.022 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542148 2026-02-22T20:37:13.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:37:13.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-22T20:37:13.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:37:13.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:13.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:13.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:37:13.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:37:13.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:37:13.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:37:13.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:37:13.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:37:13.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:37:13.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:37:13.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:37:13.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:37:13.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:13.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:13.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:37:13.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:37:13.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:37:13.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: seq 1 7 2026-02-22T20:37:13.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:37:13.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ1 2026-02-22T20:37:13.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2026-02-22T20:37:13.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:37:13.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:37:13.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2026-02-22T20:37:13.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:37:13.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:37:13.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:37:13.930 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:37:13.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:37:14.134 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:37:14.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:37:14.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:37:14.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:37:14.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 1 % 2 2026-02-22T20:37:14.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-22T20:37:14.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:37:14.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3761: corrupt_scrub_erasure: local payload=UVWXYZZZ 2026-02-22T20:37:14.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3762: corrupt_scrub_erasure: echo UVWXYZZZ 2026-02-22T20:37:14.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3763: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:37:14.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:37:14.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:37:14.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:37:14.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:37:14.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:37:14.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:37:14.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:37:14.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:37:14.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:37:14.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:37:14.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:37:14.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:37:14.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:37:14.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:37:14.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:37:14.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:37:14.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:37:14.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:37:14.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:37:14.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:37:14.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:37:14.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:37:14.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:37:14.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:37:14.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:37:14.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:37:14.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:37:14.791 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:37:14.791 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:37:14.791 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:37:14.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:14.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:14.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:37:14.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:37:14.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:37:14.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:37:14.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:37:14.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:37:14.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:37:14.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:37:14.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:37:14.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:37:14.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:37:14.793 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:37:14.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:37:14.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:37:14.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:37:14.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:37:14.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:37:14.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:37:14.812 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:14.797+0000 7f0f07d1ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:14.813 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:14.801+0000 7f0f07d1ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:14.814 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:14.801+0000 7f0f07d1ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:14.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:37:14.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:37:14.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:37:14.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:37:14.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:37:14.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:14.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:37:14.950 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:37:14.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:14.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:37:15.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:37:15.207 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:15.193+0000 7f0f07d1ba40 -1 Falling back to public interface 2026-02-22T20:37:15.827 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:15.813+0000 7f0f07d1ba40 -1 osd.1 31 log_to_monitors true 2026-02-22T20:37:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:37:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:16.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:37:16.109 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:16.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:16.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:37:16.265 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 35 up_thru 35 down_at 32 last_clean_interval [9,31) [v2:127.0.0.1:6810/3757846137,v1:127.0.0.1:6811/3757846137] [v2:127.0.0.1:6812/3757846137,v1:127.0.0.1:6813/3757846137] exists,up ddce7ffd-1827-4a8c-a1b3-20246bd6a9a4 2026-02-22T20:37:16.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:37:16.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:37:16.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:37:16.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:37:16.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:37:16.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:37:16.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:37:16.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:37:16.268 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:37:16.268 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:37:16.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:37:16.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:37:16.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:37:16.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:37:16.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:37:16.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:37:16.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:37:16.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:37:16.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:37:16.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:37:16.543 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:16.544 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:37:16.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:37:16.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:16.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:37:16.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-22T20:37:16.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-22T20:37:16.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-22T20:37:16.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:16.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:37:16.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855362 2026-02-22T20:37:16.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855362 2026-02-22T20:37:16.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-150323855362' 2026-02-22T20:37:16.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:16.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:37:16.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-22T20:37:16.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-22T20:37:16.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-150323855362 2-60129542150' 2026-02-22T20:37:16.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:16.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-22T20:37:16.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:16.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:37:16.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-22T20:37:16.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:16.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-22T20:37:16.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-22T20:37:16.766 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836488 2026-02-22T20:37:16.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:16.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-22T20:37:16.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:17.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:37:17.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:18.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-22T20:37:18.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:19.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:37:19.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:19.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-22T20:37:19.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:19.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-150323855362 2026-02-22T20:37:19.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:19.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:37:19.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-150323855362 2026-02-22T20:37:19.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:19.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855362 2026-02-22T20:37:19.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 150323855362' 2026-02-22T20:37:19.248 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 150323855362 2026-02-22T20:37:19.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:37:19.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855362 -lt 150323855362 2026-02-22T20:37:19.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:19.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-22T20:37:19.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:19.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:37:19.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-22T20:37:19.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:19.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-22T20:37:19.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-22T20:37:19.421 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542150 2026-02-22T20:37:19.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:37:19.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542151 -lt 60129542150 2026-02-22T20:37:19.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:37:19.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:19.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:19.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:37:19.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:37:19.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:37:19.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:37:19.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:37:19.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:37:19.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:37:19.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:37:19.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:37:19.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:37:19.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:19.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:37:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:37:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:37:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:37:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ2 2026-02-22T20:37:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2026-02-22T20:37:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:37:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:37:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2026-02-22T20:37:20.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:37:20.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:37:20.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:37:20.339 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:37:20.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:37:20.544 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:37:20.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:37:20.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:37:20.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:37:20.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 2 % 2 2026-02-22T20:37:20.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-22T20:37:20.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:37:20.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3768: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2026-02-22T20:37:20.578 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T20:37:20.578 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T20:37:20.578 INFO:tasks.workunit.client.0.trial150.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 8.6754e-05 s, 23.6 MB/s 2026-02-22T20:37:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3769: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:37:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:37:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:37:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:37:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:37:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:37:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:37:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:37:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:37:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:37:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:37:20.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:37:20.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:37:20.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:37:20.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:37:20.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:37:20.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:37:20.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:37:20.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:37:20.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:37:20.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:37:20.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:37:20.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:37:20.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:37:21.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:37:21.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:37:21.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:37:21.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:37:21.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:37:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:37:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:37:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:37:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:37:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:37:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:37:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:37:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:37:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:37:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:37:21.180 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:37:21.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:37:21.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:37:21.181 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:37:21.181 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:37:21.181 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:37:21.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:21.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:21.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:37:21.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:37:21.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:37:21.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:37:21.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:37:21.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:37:21.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:37:21.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:37:21.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:37:21.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:37:21.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:37:21.184 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:37:21.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:37:21.184 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:37:21.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:37:21.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:37:21.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:37:21.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:37:21.196 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:21.181+0000 7f3e770d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:21.197 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:21.185+0000 7f3e770d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:21.198 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:21.185+0000 7f3e770d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:21.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:37:21.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:37:21.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:37:21.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:37:21.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:37:21.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:21.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:37:21.346 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:37:21.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:21.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:37:21.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:37:21.611 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:21.597+0000 7f3e770d2a40 -1 Falling back to public interface 2026-02-22T20:37:22.261 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:22.249+0000 7f3e770d2a40 -1 osd.0 38 log_to_monitors true 2026-02-22T20:37:22.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:37:22.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:22.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:37:22.503 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:22.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:22.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:37:22.658 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 42 up_thru 42 down_at 39 last_clean_interval [5,38) [v2:127.0.0.1:6802/1308547885,v1:127.0.0.1:6803/1308547885] [v2:127.0.0.1:6804/1308547885,v1:127.0.0.1:6805/1308547885] exists,up 66150e60-7a54-49c2-a824-b3bc1b80d828 2026-02-22T20:37:22.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:37:22.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:37:22.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:37:22.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:37:22.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:37:22.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:37:22.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:37:22.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:37:22.660 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:37:22.660 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:37:22.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:37:22.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:37:22.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:37:22.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:37:22.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:37:22.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:37:22.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:37:22.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:37:22.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:37:22.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:37:22.932 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:22.932 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:37:22.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:37:22.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:22.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:37:23.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626434 2026-02-22T20:37:23.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626434 2026-02-22T20:37:23.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-180388626434' 2026-02-22T20:37:23.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:23.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:37:23.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=150323855364 2026-02-22T20:37:23.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 150323855364 2026-02-22T20:37:23.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-180388626434 1-150323855364' 2026-02-22T20:37:23.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:23.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:37:23.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-22T20:37:23.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-22T20:37:23.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-180388626434 1-150323855364 2-60129542153' 2026-02-22T20:37:23.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:23.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-180388626434 2026-02-22T20:37:23.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:23.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:37:23.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-180388626434 2026-02-22T20:37:23.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:23.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626434 2026-02-22T20:37:23.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 180388626434' 2026-02-22T20:37:23.156 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 180388626434 2026-02-22T20:37:23.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:23.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 180388626434 2026-02-22T20:37:23.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:24.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:37:24.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:24.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626434 -lt 180388626434 2026-02-22T20:37:24.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:24.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-150323855364 2026-02-22T20:37:24.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:24.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:37:24.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-150323855364 2026-02-22T20:37:24.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:24.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=150323855364 2026-02-22T20:37:24.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 150323855364' 2026-02-22T20:37:24.487 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 150323855364 2026-02-22T20:37:24.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:37:24.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 150323855364 -lt 150323855364 2026-02-22T20:37:24.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:24.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-22T20:37:24.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:24.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:37:24.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-22T20:37:24.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:24.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-22T20:37:24.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-22T20:37:24.657 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542153 2026-02-22T20:37:24.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:37:24.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542153 2026-02-22T20:37:24.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:37:24.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:24.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:25.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:37:25.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:37:25.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:37:25.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:37:25.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:37:25.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:37:25.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:37:25.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:37:25.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:37:25.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:37:25.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:25.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:25.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:37:25.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:37:25.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:37:25.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:37:25.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ3 2026-02-22T20:37:25.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2026-02-22T20:37:25.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:37:25.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:37:25.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2026-02-22T20:37:25.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:37:25.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:37:25.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:37:25.565 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:37:25.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:37:25.771 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:37:25.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:37:25.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:37:25.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:37:25.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 3 % 2 2026-02-22T20:37:25.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-22T20:37:25.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:37:25.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3774: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:37:25.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:37:26.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:37:26.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2026-02-22T20:37:26.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:37:26.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:37:26.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:37:26.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:37:26.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:37:26.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2026-02-22T20:37:26.297 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2026-02-22T20:37:26.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:37:26.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:37:26.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:37:26.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:37:26.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:37:26.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:37:26.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:37:26.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:37:26.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:37:26.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:37:26.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:37:26.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:37:26.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:37:26.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:37:26.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:37:26.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:37:26.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:37:26.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:37:26.417 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:37:26.417 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:37:26.417 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:37:26.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:26.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:26.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:37:26.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:37:26.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:37:26.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:37:26.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:37:26.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:37:26.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:37:26.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:37:26.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:37:26.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:37:26.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:37:26.420 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:37:26.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:37:26.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:37:26.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:37:26.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:37:26.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:37:26.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:37:26.429 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:26.413+0000 7fc560015a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:26.430 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:26.417+0000 7fc560015a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:26.431 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:26.417+0000 7fc560015a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:26.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:37:26.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:37:26.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:37:26.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:37:26.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:37:26.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:26.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:37:26.571 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:37:26.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:26.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:37:26.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:37:26.811 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:26.797+0000 7fc560015a40 -1 Falling back to public interface 2026-02-22T20:37:27.379 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:27.365+0000 7fc560015a40 -1 osd.1 46 log_to_monitors true 2026-02-22T20:37:27.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:37:27.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:27.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:37:27.732 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:27.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:27.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:37:27.891 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 50 up_thru 50 down_at 47 last_clean_interval [35,46) [v2:127.0.0.1:6810/3640968928,v1:127.0.0.1:6811/3640968928] [v2:127.0.0.1:6812/3640968928,v1:127.0.0.1:6813/3640968928] exists,up ddce7ffd-1827-4a8c-a1b3-20246bd6a9a4 2026-02-22T20:37:27.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:37:27.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:37:27.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:37:27.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:37:27.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:37:27.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:37:27.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:37:27.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:37:27.892 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:37:27.892 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:37:27.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:37:27.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:37:27.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:37:28.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:37:28.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:37:28.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:37:28.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:37:28.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:37:28.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:37:28.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:37:28.163 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:28.163 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:37:28.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:37:28.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:28.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:37:28.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=180388626436 2026-02-22T20:37:28.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 180388626436 2026-02-22T20:37:28.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-180388626436' 2026-02-22T20:37:28.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:28.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:37:28.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364802 2026-02-22T20:37:28.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364802 2026-02-22T20:37:28.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-180388626436 1-214748364802' 2026-02-22T20:37:28.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:28.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:37:28.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-22T20:37:28.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-22T20:37:28.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-180388626436 1-214748364802 2-60129542155' 2026-02-22T20:37:28.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:28.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-180388626436 2026-02-22T20:37:28.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:28.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:37:28.387 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-180388626436 2026-02-22T20:37:28.387 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:28.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=180388626436 2026-02-22T20:37:28.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 180388626436' 2026-02-22T20:37:28.389 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 180388626436 2026-02-22T20:37:28.389 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:28.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626435 -lt 180388626436 2026-02-22T20:37:28.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:29.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:37:29.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:29.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626435 -lt 180388626436 2026-02-22T20:37:29.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:30.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:37:30.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:30.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 180388626436 -lt 180388626436 2026-02-22T20:37:30.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:30.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-214748364802 2026-02-22T20:37:30.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:30.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:37:30.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-214748364802 2026-02-22T20:37:30.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:30.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=214748364802 2026-02-22T20:37:30.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 214748364802' 2026-02-22T20:37:30.885 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 214748364802 2026-02-22T20:37:30.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:37:31.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 214748364802 -lt 214748364802 2026-02-22T20:37:31.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:31.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-22T20:37:31.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:31.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:37:31.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-22T20:37:31.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:31.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-22T20:37:31.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-22T20:37:31.063 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542155 2026-02-22T20:37:31.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:37:31.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-22T20:37:31.223 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:37:31.223 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:31.224 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:31.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:37:31.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:37:31.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:37:31.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:37:31.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:37:31.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:37:31.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:37:31.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:37:31.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:37:31.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:37:31.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:31.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:31.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:37:31.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:37:31.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:37:31.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:37:31.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ4 2026-02-22T20:37:31.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2026-02-22T20:37:31.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:37:31.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:37:31.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2026-02-22T20:37:31.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:37:31.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:37:31.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:37:31.972 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:37:31.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:37:32.178 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:37:32.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:37:32.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:37:32.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:37:32.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 4 % 2 2026-02-22T20:37:32.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-22T20:37:32.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:37:32.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3778: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2026-02-22T20:37:32.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3779: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3782: corrupt_scrub_erasure: echo -n bad-val 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3783: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:37:32.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:37:32.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:37:32.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:37:32.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:37:32.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:37:32.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-22T20:37:32.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:37:32.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:37:32.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:37:32.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:37:32.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:37:32.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-22T20:37:32.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:37:32.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:37:32.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:37:32.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:37:32.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:37:32.865 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:37:32.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:37:32.866 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:37:32.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:37:32.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:37:32.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:37:32.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:37:32.885 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:32.869+0000 7fa628ed0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:32.886 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:32.873+0000 7fa628ed0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:32.887 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:32.873+0000 7fa628ed0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:33.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:37:33.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:37:33.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:37:33.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:37:33.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:37:33.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:33.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:37:33.029 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:37:33.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:33.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:37:33.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:37:33.291 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:33.277+0000 7fa628ed0a40 -1 Falling back to public interface 2026-02-22T20:37:33.908 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:33.893+0000 7fa628ed0a40 -1 osd.0 54 log_to_monitors true 2026-02-22T20:37:34.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:37:34.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:34.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:37:34.190 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:34.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:34.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:37:34.343 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 58 up_thru 58 down_at 55 last_clean_interval [42,54) [v2:127.0.0.1:6802/345153629,v1:127.0.0.1:6803/345153629] [v2:127.0.0.1:6804/345153629,v1:127.0.0.1:6805/345153629] exists,up 66150e60-7a54-49c2-a824-b3bc1b80d828 2026-02-22T20:37:34.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:37:34.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:37:34.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:37:34.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:37:34.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:37:34.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:37:34.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:37:34.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:37:34.345 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:37:34.345 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:37:34.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:37:34.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:37:34.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:37:34.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:37:34.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:37:34.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:37:34.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:37:34.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:37:34.459 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:37:34.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:37:34.617 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:34.617 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:37:34.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:37:34.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:34.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:37:34.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103170 2026-02-22T20:37:34.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103170 2026-02-22T20:37:34.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103170' 2026-02-22T20:37:34.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:34.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:37:34.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364804 2026-02-22T20:37:34.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364804 2026-02-22T20:37:34.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103170 1-214748364804' 2026-02-22T20:37:34.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:34.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:37:34.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542157 2026-02-22T20:37:34.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542157 2026-02-22T20:37:34.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103170 1-214748364804 2-60129542157' 2026-02-22T20:37:34.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:34.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-249108103170 2026-02-22T20:37:34.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:34.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:37:34.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-249108103170 2026-02-22T20:37:34.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:34.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=249108103170 2026-02-22T20:37:34.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 249108103170' 2026-02-22T20:37:34.837 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 249108103170 2026-02-22T20:37:34.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:35.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 249108103170 2026-02-22T20:37:35.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:36.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:37:36.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:36.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103170 -lt 249108103170 2026-02-22T20:37:36.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:36.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-214748364804 2026-02-22T20:37:36.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:36.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:37:36.167 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-214748364804 2026-02-22T20:37:36.167 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:36.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=214748364804 2026-02-22T20:37:36.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 214748364804' 2026-02-22T20:37:36.169 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 214748364804 2026-02-22T20:37:36.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:37:36.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 214748364804 -lt 214748364804 2026-02-22T20:37:36.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:36.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542157 2026-02-22T20:37:36.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:36.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:37:36.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542157 2026-02-22T20:37:36.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:36.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542157 2026-02-22T20:37:36.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2026-02-22T20:37:36.338 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542157 2026-02-22T20:37:36.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:37:36.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542157 -lt 60129542157 2026-02-22T20:37:36.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:37:36.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:36.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:36.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:37:36.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:37:36.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:37:36.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:37:36.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:37:36.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:37:36.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:37:36.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:37:36.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:37:36.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:37:36.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:36.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:37.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:37:37.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:37:37.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:37:37.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3784: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-22T20:37:37.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:37:37.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:37:37.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T20:37:37.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:37:37.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-22T20:37:37.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:37:37.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:37:37.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T20:37:37.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:37:37.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-22T20:37:37.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:37:37.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:37:37.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:37:37.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:37:37.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:37:37.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:37:37.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-22T20:37:37.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:37:37.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:37:37.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T20:37:37.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:37:37.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:37:37.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-22T20:37:37.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-22T20:37:37.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:37:37.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:37:37.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:37:37.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:37:37.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:37:37.678 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:37:37.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:37.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:37.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:37:37.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:37:37.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:37:37.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:37:37.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:37:37.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:37:37.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:37:37.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:37:37.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:37:37.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:37:37.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:37:37.680 INFO:tasks.workunit.client.0.trial150.stderr:start osd.2 2026-02-22T20:37:37.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:37:37.680 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-22T20:37:37.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:37:37.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:37:37.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:37:37.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:37:37.699 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:37.686+0000 7f2e8553ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:37.701 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:37.690+0000 7f2e8553ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:37.702 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:37.690+0000 7f2e8553ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:37.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:37:37.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:37:37.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:37:37.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:37:37.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:37:37.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:37.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:37:37.842 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:37:37.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:37.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:37:38.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:37:38.086 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:38.074+0000 7f2e8553ca40 -1 Falling back to public interface 2026-02-22T20:37:38.734 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:38.722+0000 7f2e8553ca40 -1 osd.2 60 log_to_monitors true 2026-02-22T20:37:39.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:37:39.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:39.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:37:39.005 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:39.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:39.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:37:39.157 INFO:tasks.workunit.client.0.trial150.stderr:osd.2 up in weight 1 up_from 63 up_thru 63 down_at 60 last_clean_interval [14,60) [v2:127.0.0.1:6818/2764485141,v1:127.0.0.1:6819/2764485141] [v2:127.0.0.1:6820/2764485141,v1:127.0.0.1:6821/2764485141] exists,up cee891b2-6c0d-496a-b376-597b5a64bc9c 2026-02-22T20:37:39.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:37:39.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:37:39.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:37:39.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:37:39.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:37:39.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:37:39.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:37:39.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:37:39.159 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:37:39.159 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:37:39.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:37:39.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:37:39.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:37:39.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:37:39.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:37:39.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:37:39.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:37:39.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:37:39.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:37:39.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:37:39.426 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:39.426 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:37:39.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:37:39.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:39.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:37:39.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103172 2026-02-22T20:37:39.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103172 2026-02-22T20:37:39.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103172' 2026-02-22T20:37:39.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:39.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:37:39.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364806 2026-02-22T20:37:39.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364806 2026-02-22T20:37:39.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103172 1-214748364806' 2026-02-22T20:37:39.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:39.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:37:39.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=270582939650 2026-02-22T20:37:39.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 270582939650 2026-02-22T20:37:39.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103172 1-214748364806 2-270582939650' 2026-02-22T20:37:39.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:39.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-249108103172 2026-02-22T20:37:39.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:39.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:37:39.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-249108103172 2026-02-22T20:37:39.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:39.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=249108103172 2026-02-22T20:37:39.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 249108103172' 2026-02-22T20:37:39.648 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 249108103172 2026-02-22T20:37:39.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:39.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103170 -lt 249108103172 2026-02-22T20:37:39.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:40.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:37:40.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:40.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103172 -lt 249108103172 2026-02-22T20:37:40.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:40.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-214748364806 2026-02-22T20:37:40.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:40.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:37:40.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-214748364806 2026-02-22T20:37:40.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:40.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=214748364806 2026-02-22T20:37:40.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 214748364806' 2026-02-22T20:37:40.982 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 214748364806 2026-02-22T20:37:40.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:37:41.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 214748364806 -lt 214748364806 2026-02-22T20:37:41.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:41.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-270582939650 2026-02-22T20:37:41.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:41.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:37:41.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-270582939650 2026-02-22T20:37:41.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:41.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=270582939650 2026-02-22T20:37:41.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 270582939650' 2026-02-22T20:37:41.148 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 270582939650 2026-02-22T20:37:41.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:37:41.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 270582939650 -lt 270582939650 2026-02-22T20:37:41.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:37:41.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:41.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:41.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:37:41.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:37:41.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:37:41.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:37:41.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:37:41.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:37:41.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:37:41.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:37:41.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:37:41.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:37:41.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:41.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:41.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:37:41.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:37:41.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:37:41.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3785: corrupt_scrub_erasure: echo -n val3-EOBJ4 2026-02-22T20:37:41.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3786: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-22T20:37:41.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:37:41.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:37:41.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T20:37:41.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:37:41.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-22T20:37:41.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:37:41.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:37:41.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T20:37:41.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:37:41.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-22T20:37:41.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:37:41.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:37:41.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:37:41.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:37:41.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:37:42.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:37:42.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-22T20:37:42.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:37:42.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:37:42.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T20:37:42.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:37:42.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:37:42.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:37:42.434 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:37:42.435 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:37:42.435 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:37:42.435 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:37:42.436 INFO:tasks.workunit.client.0.trial150.stderr:start osd.2 2026-02-22T20:37:42.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:37:42.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-22T20:37:42.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:37:42.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:37:42.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:37:42.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:37:42.460 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:42.446+0000 7f9372c74a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:42.462 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:42.450+0000 7f9372c74a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:42.463 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:42.450+0000 7f9372c74a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:42.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:37:42.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:37:42.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:37:42.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:37:42.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:37:42.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:42.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:37:42.597 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:37:42.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:42.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:37:42.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:37:42.854 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:42.842+0000 7f9372c74a40 -1 Falling back to public interface 2026-02-22T20:37:43.497 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:43.486+0000 7f9372c74a40 -1 osd.2 65 log_to_monitors true 2026-02-22T20:37:43.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:37:43.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:43.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:37:43.757 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:43.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:43.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:37:43.917 INFO:tasks.workunit.client.0.trial150.stderr:osd.2 up in weight 1 up_from 69 up_thru 69 down_at 66 last_clean_interval [63,65) [v2:127.0.0.1:6818/3847280453,v1:127.0.0.1:6819/3847280453] [v2:127.0.0.1:6820/3847280453,v1:127.0.0.1:6821/3847280453] exists,up cee891b2-6c0d-496a-b376-597b5a64bc9c 2026-02-22T20:37:43.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:37:43.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:37:43.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:37:43.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:37:43.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:37:43.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:37:43.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:37:43.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:37:43.918 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:37:43.918 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:37:43.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:37:43.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:37:43.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:37:44.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:37:44.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:37:44.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:37:44.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:37:44.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:37:44.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:37:44.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:37:44.195 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:44.195 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:37:44.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:37:44.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:44.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:37:44.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103174 2026-02-22T20:37:44.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103174 2026-02-22T20:37:44.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103174' 2026-02-22T20:37:44.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:44.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:37:44.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=214748364808 2026-02-22T20:37:44.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 214748364808 2026-02-22T20:37:44.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103174 1-214748364808' 2026-02-22T20:37:44.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:44.341 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:37:44.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743426 2026-02-22T20:37:44.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743426 2026-02-22T20:37:44.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103174 1-214748364808 2-296352743426' 2026-02-22T20:37:44.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:44.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-249108103174 2026-02-22T20:37:44.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:44.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:37:44.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-249108103174 2026-02-22T20:37:44.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:44.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=249108103174 2026-02-22T20:37:44.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 249108103174' 2026-02-22T20:37:44.416 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 249108103174 2026-02-22T20:37:44.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:44.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103173 -lt 249108103174 2026-02-22T20:37:44.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:45.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:37:45.578 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:45.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103173 -lt 249108103174 2026-02-22T20:37:45.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:46.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:37:46.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:46.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103174 -lt 249108103174 2026-02-22T20:37:46.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:46.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-214748364808 2026-02-22T20:37:46.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:46.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:37:46.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-214748364808 2026-02-22T20:37:46.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:46.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=214748364808 2026-02-22T20:37:46.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 214748364808' 2026-02-22T20:37:46.908 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 214748364808 2026-02-22T20:37:46.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:37:47.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 214748364808 -lt 214748364808 2026-02-22T20:37:47.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:47.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743426 2026-02-22T20:37:47.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:47.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:37:47.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743426 2026-02-22T20:37:47.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:47.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743426 2026-02-22T20:37:47.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743426' 2026-02-22T20:37:47.077 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 296352743426 2026-02-22T20:37:47.078 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:37:47.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743426 -lt 296352743426 2026-02-22T20:37:47.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:37:47.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:47.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:47.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:37:47.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:37:47.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:37:47.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:37:47.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:37:47.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:37:47.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:37:47.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:37:47.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:37:47.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:37:47.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:47.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:47.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:37:47.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:37:47.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:37:47.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3787: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2026-02-22T20:37:47.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:37:47.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ5 2026-02-22T20:37:47.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2026-02-22T20:37:47.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:37:47.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:37:47.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2026-02-22T20:37:47.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:37:47.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:37:47.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:37:48.010 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:37:48.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:37:48.216 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:37:48.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:37:48.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:37:48.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:37:48.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 5 % 2 2026-02-22T20:37:48.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-22T20:37:48.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:37:48.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3792: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2026-02-22T20:37:48.253 INFO:tasks.workunit.client.0.trial150.stderr:2+0 records in 2026-02-22T20:37:48.253 INFO:tasks.workunit.client.0.trial150.stderr:2+0 records out 2026-02-22T20:37:48.253 INFO:tasks.workunit.client.0.trial150.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 7.8107e-05 s, 52.4 MB/s 2026-02-22T20:37:48.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3793: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:37:48.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:37:48.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:37:48.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:37:48.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:37:48.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:37:48.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:37:48.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:37:48.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:37:48.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:37:48.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:37:48.871 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:37:48.871 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:37:48.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:37:48.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:37:48.873 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:37:48.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:37:48.873 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:37:48.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:37:48.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:37:48.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:37:48.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:37:48.895 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:48.882+0000 7f5245264a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:48.897 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:48.882+0000 7f5245264a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:48.897 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:48.886+0000 7f5245264a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:49.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:37:49.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:37:49.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:37:49.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:37:49.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:37:49.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:49.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:37:49.030 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:37:49.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:49.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:37:49.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:37:49.235 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:49.222+0000 7f5245264a40 -1 Falling back to public interface 2026-02-22T20:37:49.778 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:49.766+0000 7f5245264a40 -1 osd.1 73 log_to_monitors true 2026-02-22T20:37:50.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:37:50.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:50.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:37:50.194 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:50.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:50.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:37:50.349 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [50,73) [v2:127.0.0.1:6810/798477159,v1:127.0.0.1:6811/798477159] [v2:127.0.0.1:6812/798477159,v1:127.0.0.1:6813/798477159] exists,up ddce7ffd-1827-4a8c-a1b3-20246bd6a9a4 2026-02-22T20:37:50.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:37:50.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:37:50.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:37:50.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:37:50.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:37:50.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:37:50.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:37:50.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:37:50.352 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:37:50.352 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:37:50.352 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:37:50.352 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:37:50.352 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:37:50.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:37:50.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:37:50.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:37:50.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:37:50.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:37:50.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:37:50.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:37:50.610 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:50.610 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:37:50.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:37:50.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:50.610 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:37:50.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=249108103176 2026-02-22T20:37:50.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 249108103176 2026-02-22T20:37:50.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103176' 2026-02-22T20:37:50.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:50.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:37:50.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481794 2026-02-22T20:37:50.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481794 2026-02-22T20:37:50.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103176 1-330712481794' 2026-02-22T20:37:50.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:50.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:37:50.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743428 2026-02-22T20:37:50.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743428 2026-02-22T20:37:50.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-249108103176 1-330712481794 2-296352743428' 2026-02-22T20:37:50.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:50.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-249108103176 2026-02-22T20:37:50.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:50.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:37:50.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-249108103176 2026-02-22T20:37:50.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:50.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=249108103176 2026-02-22T20:37:50.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 249108103176' 2026-02-22T20:37:50.839 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 249108103176 2026-02-22T20:37:50.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:51.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103175 -lt 249108103176 2026-02-22T20:37:51.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:52.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:37:52.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:52.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 249108103176 -lt 249108103176 2026-02-22T20:37:52.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:52.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-330712481794 2026-02-22T20:37:52.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:52.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:37:52.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-330712481794 2026-02-22T20:37:52.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:52.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=330712481794 2026-02-22T20:37:52.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2026-02-22T20:37:52.172 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 330712481794 2026-02-22T20:37:52.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:37:52.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481794 -lt 330712481794 2026-02-22T20:37:52.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:52.342 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743428 2026-02-22T20:37:52.342 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:52.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:37:52.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743428 2026-02-22T20:37:52.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:52.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743428 2026-02-22T20:37:52.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743428' 2026-02-22T20:37:52.346 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 296352743428 2026-02-22T20:37:52.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:37:52.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743428 -lt 296352743428 2026-02-22T20:37:52.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:37:52.511 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:52.511 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:52.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:37:52.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:37:52.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:37:52.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:37:52.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:37:52.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:37:52.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:37:52.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:37:52.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:37:52.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:37:52.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:52.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:53.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:37:53.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:37:53.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:37:53.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:37:53.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ6 2026-02-22T20:37:53.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2026-02-22T20:37:53.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:37:53.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:37:53.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2026-02-22T20:37:53.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:37:53.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:37:53.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:37:53.253 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:37:53.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:37:53.458 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:37:53.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:37:53.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:37:53.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:37:53.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 6 % 2 2026-02-22T20:37:53.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-22T20:37:53.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:37:53.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3797: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-22T20:37:53.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:37:53.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:37:53.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:37:53.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:37:53.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-22T20:37:53.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:37:53.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:37:53.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:37:53.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:37:53.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:37:53.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:37:53.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:37:53.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:37:53.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:37:53.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:37:53.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:37:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-22T20:37:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:37:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:37:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:37:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:37:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:37:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2026-02-22T20:37:54.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:37:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:37:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:37:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:37:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:37:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:37:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:37:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:37:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:37:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:54.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:37:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:37:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:37:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:37:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:37:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:37:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:37:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:37:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:37:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:37:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:37:54.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:37:54.088 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:37:54.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:37:54.088 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:37:54.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:37:54.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:37:54.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:37:54.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:37:54.106 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:54.094+0000 7f07003b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:54.108 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:54.094+0000 7f07003b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:54.109 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:54.098+0000 7f07003b4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:37:54.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:37:54.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:37:54.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:37:54.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:37:54.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:37:54.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:54.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:37:54.245 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:37:54.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:54.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:37:54.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:37:54.538 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:54.526+0000 7f07003b4a40 -1 Falling back to public interface 2026-02-22T20:37:55.222 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:37:55.210+0000 7f07003b4a40 -1 osd.0 80 log_to_monitors true 2026-02-22T20:37:55.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:37:55.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:37:55.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:37:55.405 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:55.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:37:55.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:37:55.566 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [58,80) [v2:127.0.0.1:6802/3127383734,v1:127.0.0.1:6803/3127383734] [v2:127.0.0.1:6804/3127383734,v1:127.0.0.1:6805/3127383734] exists,up 66150e60-7a54-49c2-a824-b3bc1b80d828 2026-02-22T20:37:55.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:37:55.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:37:55.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:37:55.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:37:55.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:37:55.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:37:55.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:37:55.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:37:55.568 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:37:55.568 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:37:55.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:37:55.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:37:55.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:37:55.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:37:55.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:37:55.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:37:55.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:37:55.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:37:55.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:37:55.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:37:55.848 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:37:55.848 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:37:55.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:37:55.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:55.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:37:55.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=360777252866 2026-02-22T20:37:55.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 360777252866 2026-02-22T20:37:55.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866' 2026-02-22T20:37:55.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:55.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:37:55.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481796 2026-02-22T20:37:55.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481796 2026-02-22T20:37:55.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866 1-330712481796' 2026-02-22T20:37:55.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:37:55.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:37:56.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743430 2026-02-22T20:37:56.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743430 2026-02-22T20:37:56.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866 1-330712481796 2-296352743430' 2026-02-22T20:37:56.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:56.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-360777252866 2026-02-22T20:37:56.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:56.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:37:56.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-360777252866 2026-02-22T20:37:56.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:56.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=360777252866 2026-02-22T20:37:56.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2026-02-22T20:37:56.068 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 360777252866 2026-02-22T20:37:56.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:37:56.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252866 -lt 360777252866 2026-02-22T20:37:56.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:56.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-330712481796 2026-02-22T20:37:56.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:56.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:37:56.229 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-330712481796 2026-02-22T20:37:56.229 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:56.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=330712481796 2026-02-22T20:37:56.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 330712481796' 2026-02-22T20:37:56.230 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 330712481796 2026-02-22T20:37:56.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:37:56.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481796 -lt 330712481796 2026-02-22T20:37:56.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:37:56.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743430 2026-02-22T20:37:56.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:37:56.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:37:56.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743430 2026-02-22T20:37:56.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:37:56.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743430 2026-02-22T20:37:56.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743430' 2026-02-22T20:37:56.404 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 296352743430 2026-02-22T20:37:56.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:37:56.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743429 -lt 296352743430 2026-02-22T20:37:56.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:57.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:37:57.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:37:57.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743429 -lt 296352743430 2026-02-22T20:37:57.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:37:58.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:37:58.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:37:58.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743430 -lt 296352743430 2026-02-22T20:37:58.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:37:58.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:58.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:59.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:37:59.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:37:59.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:37:59.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:37:59.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:37:59.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:37:59.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:37:59.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:37:59.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:37:59.254 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:37:59.254 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:37:59.254 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3798: corrupt_scrub_erasure: echo -n bad-val 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3799: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:37:59.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:37:59.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:37:59.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:37:59.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:37:59.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:37:59.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:37:59.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:37:59.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-22T20:37:59.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:37:59.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:37:59.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:37:59.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:37:59.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:37:59.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-22T20:38:00.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:38:00.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:38:00.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:38:00.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:38:00.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:38:00.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:38:00.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:38:00.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:38:00.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:38:00.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:38:00.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:38:00.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:38:00.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:38:00.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:38:00.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:38:00.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:38:00.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:38:00.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:38:00.046 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:38:00.046 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:00.046 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:00.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:00.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:00.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:38:00.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:38:00.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:38:00.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:38:00.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:38:00.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:38:00.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:38:00.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:38:00.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:38:00.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:38:00.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:38:00.048 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:38:00.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:38:00.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:38:00.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:38:00.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:38:00.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:38:00.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:38:00.063 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:00.050+0000 7f0478f92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:00.065 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:00.050+0000 7f0478f92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:00.066 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:00.054+0000 7f0478f92a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:00.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:38:00.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:38:00.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:38:00.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:38:00.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:38:00.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:00.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:38:00.207 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:38:00.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:00.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:38:00.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:38:00.490 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:00.478+0000 7f0478f92a40 -1 Falling back to public interface 2026-02-22T20:38:01.169 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:01.158+0000 7f0478f92a40 -1 osd.1 85 log_to_monitors true 2026-02-22T20:38:01.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:38:01.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:01.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:38:01.365 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:01.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:01.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:38:01.524 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 86 last_clean_interval [77,85) [v2:127.0.0.1:6810/762120797,v1:127.0.0.1:6811/762120797] [v2:127.0.0.1:6812/762120797,v1:127.0.0.1:6813/762120797] exists,up ddce7ffd-1827-4a8c-a1b3-20246bd6a9a4 2026-02-22T20:38:01.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:38:01.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:38:01.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:38:01.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:38:01.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:38:01.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:38:01.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:38:01.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:38:01.526 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:38:01.526 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:38:01.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:38:01.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:38:01.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:38:01.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:38:01.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:38:01.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:38:01.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:38:01.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:38:01.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:38:01.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:38:01.801 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:01.801 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:38:01.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:38:01.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:01.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:38:01.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=360777252868 2026-02-22T20:38:01.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 360777252868 2026-02-22T20:38:01.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252868' 2026-02-22T20:38:01.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:01.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:38:01.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089346 2026-02-22T20:38:01.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089346 2026-02-22T20:38:01.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252868 1-382252089346' 2026-02-22T20:38:01.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:01.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:38:02.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743432 2026-02-22T20:38:02.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743432 2026-02-22T20:38:02.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252868 1-382252089346 2-296352743432' 2026-02-22T20:38:02.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:02.019 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-360777252868 2026-02-22T20:38:02.019 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:02.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:38:02.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-360777252868 2026-02-22T20:38:02.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:02.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=360777252868 2026-02-22T20:38:02.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 360777252868' 2026-02-22T20:38:02.023 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 360777252868 2026-02-22T20:38:02.023 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:02.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252868 -lt 360777252868 2026-02-22T20:38:02.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:02.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089346 2026-02-22T20:38:02.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:02.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:38:02.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089346 2026-02-22T20:38:02.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089346 2026-02-22T20:38:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2026-02-22T20:38:02.196 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 382252089346 2026-02-22T20:38:02.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:38:02.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089346 -lt 382252089346 2026-02-22T20:38:02.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:02.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743432 2026-02-22T20:38:02.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:02.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:38:02.368 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743432 2026-02-22T20:38:02.368 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:02.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743432 2026-02-22T20:38:02.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743432' 2026-02-22T20:38:02.370 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 296352743432 2026-02-22T20:38:02.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:38:02.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743432 -lt 296352743432 2026-02-22T20:38:02.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:38:02.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:38:02.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:38:02.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:38:02.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:38:02.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:38:02.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:38:02.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:38:02.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:38:02.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:38:02.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:38:02.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:38:02.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:38:02.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:38:02.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:38:03.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:38:03.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:38:03.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:38:03.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:38:03.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ7 2026-02-22T20:38:03.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2026-02-22T20:38:03.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:38:03.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:38:03.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2026-02-22T20:38:03.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:38:03.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:38:03.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:38:03.288 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:38:03.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:38:03.493 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:38:03.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:38:03.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:38:03.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:38:03.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 7 % 2 2026-02-22T20:38:03.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-22T20:38:03.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:38:03.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3803: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2026-02-22T20:38:03.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3804: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2026-02-22T20:38:03.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3805: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2026-02-22T20:38:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3808: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-22T20:38:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:38:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:38:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:38:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:38:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-22T20:38:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:38:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:38:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:38:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:38:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:38:03.548 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:38:03.548 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:38:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:38:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:38:03.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:38:03.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:38:03.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-22T20:38:03.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:38:03.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:38:03.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:38:03.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:38:03.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:38:03.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2026-02-22T20:38:04.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:38:04.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:38:04.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:38:04.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:38:04.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:38:04.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:38:04.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:38:04.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:38:04.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:38:04.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:38:04.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:38:04.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:38:04.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:38:04.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:38:04.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:38:04.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:38:04.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:38:04.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:38:04.033 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:38:04.034 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:04.034 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:04.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:04.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:04.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:38:04.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:38:04.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:38:04.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:38:04.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:38:04.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:38:04.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:38:04.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:38:04.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:38:04.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:38:04.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:38:04.035 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:38:04.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:38:04.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:38:04.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:38:04.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:38:04.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:38:04.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:38:04.052 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:04.038+0000 7f7574b15a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:04.053 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:04.042+0000 7f7574b15a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:04.054 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:04.042+0000 7f7574b15a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:04.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:38:04.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:38:04.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:38:04.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:38:04.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:38:04.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:04.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:38:04.197 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:38:04.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:04.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:38:04.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:38:04.455 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:04.442+0000 7f7574b15a40 -1 Falling back to public interface 2026-02-22T20:38:05.056 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:05.042+0000 7f7574b15a40 -1 osd.0 92 log_to_monitors true 2026-02-22T20:38:05.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:38:05.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:05.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:38:05.359 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:05.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:05.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:38:05.515 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 96 up_thru 96 down_at 93 last_clean_interval [84,92) [v2:127.0.0.1:6802/3375028207,v1:127.0.0.1:6803/3375028207] [v2:127.0.0.1:6804/3375028207,v1:127.0.0.1:6805/3375028207] exists,up 66150e60-7a54-49c2-a824-b3bc1b80d828 2026-02-22T20:38:05.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:38:05.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:38:05.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:38:05.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:38:05.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:38:05.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:38:05.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:38:05.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:38:05.517 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:38:05.517 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:38:05.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:38:05.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:38:05.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:38:05.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:38:05.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:38:05.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:38:05.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:38:05.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:38:05.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:38:05.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:38:05.795 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:05.795 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:38:05.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:38:05.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:05.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:38:05.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860418 2026-02-22T20:38:05.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860418 2026-02-22T20:38:05.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-412316860418' 2026-02-22T20:38:05.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:05.866 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:38:05.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089347 2026-02-22T20:38:05.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089347 2026-02-22T20:38:05.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-412316860418 1-382252089347' 2026-02-22T20:38:05.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:05.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:38:06.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743434 2026-02-22T20:38:06.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743434 2026-02-22T20:38:06.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-412316860418 1-382252089347 2-296352743434' 2026-02-22T20:38:06.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:06.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-412316860418 2026-02-22T20:38:06.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:06.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:38:06.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-412316860418 2026-02-22T20:38:06.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:06.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860418 2026-02-22T20:38:06.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 412316860418' 2026-02-22T20:38:06.014 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 412316860418 2026-02-22T20:38:06.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:06.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860418 -lt 412316860418 2026-02-22T20:38:06.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:06.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089347 2026-02-22T20:38:06.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:06.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:38:06.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089347 2026-02-22T20:38:06.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:06.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089347 2026-02-22T20:38:06.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089347' 2026-02-22T20:38:06.180 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 382252089347 2026-02-22T20:38:06.180 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:38:06.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089347 -lt 382252089347 2026-02-22T20:38:06.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:06.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743434 2026-02-22T20:38:06.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:06.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:38:06.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743434 2026-02-22T20:38:06.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:06.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743434 2026-02-22T20:38:06.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743434' 2026-02-22T20:38:06.341 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 296352743434 2026-02-22T20:38:06.342 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:38:06.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743434 -lt 296352743434 2026-02-22T20:38:06.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:38:06.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:38:06.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:38:06.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:38:06.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:38:06.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:38:06.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:38:06.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:38:06.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:38:06.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:38:06.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:38:06.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:38:06.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:38:06.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:38:06.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:38:07.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:38:07.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:38:07.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:38:07.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3809: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-22T20:38:07.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:38:07.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:38:07.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:38:07.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:38:07.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-22T20:38:07.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:38:07.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:38:07.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:38:07.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:38:07.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:38:07.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:38:07.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:38:07.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:38:07.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:38:07.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:38:07.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:38:07.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-22T20:38:07.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:38:07.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:38:07.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:38:07.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:38:07.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:38:07.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-22T20:38:07.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:38:07.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:38:07.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:38:07.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:38:07.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:38:07.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:38:07.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:38:07.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:38:07.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:38:07.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:38:07.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:38:07.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:38:07.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:38:07.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:38:07.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:38:07.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:38:07.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:38:07.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:38:07.602 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:38:07.602 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:07.602 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:07.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:07.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:07.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:38:07.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:38:07.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:38:07.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:38:07.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:38:07.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:38:07.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:38:07.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:38:07.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:38:07.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:38:07.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:38:07.604 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:38:07.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:38:07.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:38:07.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:38:07.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:38:07.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:38:07.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:38:07.624 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:07.610+0000 7f873d45fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:07.626 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:07.614+0000 7f873d45fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:07.627 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:07.614+0000 7f873d45fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:07.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:38:07.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:38:07.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:38:07.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:38:07.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:38:07.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:07.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:38:07.764 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:38:07.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:07.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:38:07.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:38:08.023 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:08.010+0000 7f873d45fa40 -1 Falling back to public interface 2026-02-22T20:38:08.554 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:08.542+0000 7f873d45fa40 -1 osd.0 97 log_to_monitors true 2026-02-22T20:38:08.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:38:08.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:08.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:38:08.925 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:08.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:08.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:38:09.083 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 101 up_thru 101 down_at 98 last_clean_interval [96,97) [v2:127.0.0.1:6802/4150290971,v1:127.0.0.1:6803/4150290971] [v2:127.0.0.1:6804/4150290971,v1:127.0.0.1:6805/4150290971] exists,up 66150e60-7a54-49c2-a824-b3bc1b80d828 2026-02-22T20:38:09.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:38:09.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:38:09.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:38:09.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:38:09.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:38:09.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:38:09.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:38:09.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:38:09.084 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:38:09.084 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:38:09.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:38:09.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:38:09.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:38:09.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:38:09.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:38:09.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:38:09.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:38:09.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:38:09.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:38:09.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:38:09.361 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:09.361 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:38:09.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:38:09.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:09.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:38:09.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696898 2026-02-22T20:38:09.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696898 2026-02-22T20:38:09.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696898' 2026-02-22T20:38:09.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:09.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:38:09.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089349 2026-02-22T20:38:09.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089349 2026-02-22T20:38:09.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696898 1-382252089349' 2026-02-22T20:38:09.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:09.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:38:09.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743436 2026-02-22T20:38:09.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743436 2026-02-22T20:38:09.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696898 1-382252089349 2-296352743436' 2026-02-22T20:38:09.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:09.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-433791696898 2026-02-22T20:38:09.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:09.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:38:09.578 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-433791696898 2026-02-22T20:38:09.578 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:09.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696898 2026-02-22T20:38:09.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 433791696898' 2026-02-22T20:38:09.580 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 433791696898 2026-02-22T20:38:09.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:09.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860418 -lt 433791696898 2026-02-22T20:38:09.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:38:10.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:38:10.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:10.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696898 -lt 433791696898 2026-02-22T20:38:10.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:10.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089349 2026-02-22T20:38:10.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:10.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:38:10.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089349 2026-02-22T20:38:10.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:10.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089349 2026-02-22T20:38:10.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089349' 2026-02-22T20:38:10.898 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 382252089349 2026-02-22T20:38:10.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:38:11.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089349 -lt 382252089349 2026-02-22T20:38:11.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:11.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743436 2026-02-22T20:38:11.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:11.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:38:11.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743436 2026-02-22T20:38:11.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:11.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743436 2026-02-22T20:38:11.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743436' 2026-02-22T20:38:11.069 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 296352743436 2026-02-22T20:38:11.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:38:11.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743436 -lt 296352743436 2026-02-22T20:38:11.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:38:11.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:38:11.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:38:11.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:38:11.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:38:11.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:38:11.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:38:11.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:38:11.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:38:11.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:38:11.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:38:11.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:38:11.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:38:11.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:38:11.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:38:11.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:38:11.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:38:11.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:38:11.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3810: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2026-02-22T20:38:11.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3816: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2026-02-22T20:38:11.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:38:11.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2026-02-22T20:38:11.767 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2026-02-22T20:38:11.767 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:38:11.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3816: corrupt_scrub_erasure: local pg=3.0 2026-02-22T20:38:11.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3818: corrupt_scrub_erasure: pg_scrub 3.0 2026-02-22T20:38:11.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-22T20:38:11.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-22T20:38:11.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:38:11.927 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:38:11.928 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:38:11.928 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:38:11.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:38:11.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:38:11.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:38:12.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:38:12.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:38:12.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:38:12.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:38:12.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:38:12.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:38:12.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:38:12.373 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:12.373 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:38:12.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:38:12.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:12.374 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:38:12.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696899 2026-02-22T20:38:12.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696899 2026-02-22T20:38:12.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696899' 2026-02-22T20:38:12.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:12.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:38:12.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089351 2026-02-22T20:38:12.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089351 2026-02-22T20:38:12.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696899 1-382252089351' 2026-02-22T20:38:12.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:12.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:38:12.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743437 2026-02-22T20:38:12.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743437 2026-02-22T20:38:12.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696899 1-382252089351 2-296352743437' 2026-02-22T20:38:12.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:12.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-433791696899 2026-02-22T20:38:12.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:12.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:38:12.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-433791696899 2026-02-22T20:38:12.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:12.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696899 2026-02-22T20:38:12.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 433791696899' 2026-02-22T20:38:12.595 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 433791696899 2026-02-22T20:38:12.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:12.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696898 -lt 433791696899 2026-02-22T20:38:12.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:38:13.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:38:13.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:13.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696898 -lt 433791696899 2026-02-22T20:38:13.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:38:14.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:38:14.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:15.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696900 -lt 433791696899 2026-02-22T20:38:15.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:15.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089351 2026-02-22T20:38:15.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:15.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:38:15.088 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089351 2026-02-22T20:38:15.088 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:15.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089351 2026-02-22T20:38:15.090 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 382252089351 2026-02-22T20:38:15.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089351' 2026-02-22T20:38:15.090 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:38:15.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089351 -lt 382252089351 2026-02-22T20:38:15.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:15.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743437 2026-02-22T20:38:15.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:15.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:38:15.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743437 2026-02-22T20:38:15.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:15.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743437 2026-02-22T20:38:15.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743437' 2026-02-22T20:38:15.260 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 296352743437 2026-02-22T20:38:15.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:38:15.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743438 -lt 296352743437 2026-02-22T20:38:15.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:38:15.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:38:15.430 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:38:15.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:38:15.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:38:15.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:38:15.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:38:15.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:38:15.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T20:38:15.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:38:15.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:38:15.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:38:15.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-22T20:38:15.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:38:15.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:38:15.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:38:15.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:38:15.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:15.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-22T20:38:15.809 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0s0 on osd.1 to scrub 2026-02-22T20:38:15.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:15.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:38:15.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:15.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:38:15.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:38:15.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:38:15.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:38:15.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:38:15.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:38:15.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:38:15.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:38:15.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:37:06.108452+0000 '>' 2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:15.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:38:16.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:38:16.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:38:16.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:38:16.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:38:16.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:38:16.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:38:16.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:38:17.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:37:06.108452+0000 '>' 2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:17.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:38:18.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:38:18.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:38:18.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:38:18.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:38:18.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:38:18.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:38:18.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:38:18.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:37:06.108452+0000 '>' 2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:18.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:38:19.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:38:19.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:38:19.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:38:19.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:38:19.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:38:19.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:38:19.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:38:19.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:37:06.108452+0000 '>' 2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:19.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:38:20.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:38:20.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:38:20.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:38:20.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:38:20.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:38:20.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:38:20.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:38:20.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:37:06.108452+0000 '>' 2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:20.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:38:21.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:38:21.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:38:21.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:38:21.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:38:21.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:38:21.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:38:21.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:38:21.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:37:06.108452+0000 '>' 2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:21.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:38:22.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:38:22.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:38:22.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:38:22.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:38:22.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:38:22.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:38:22.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:38:22.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:16.277900+0000 '>' 2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:22.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:38:22.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3820: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2026-02-22T20:38:22.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3822: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2026-02-22T20:38:22.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3822: corrupt_scrub_erasure: test 1 = 1 2026-02-22T20:38:22.987 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3824: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-22T20:38:22.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3824: corrupt_scrub_erasure: test 3.0 = 3.0 2026-02-22T20:38:22.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3826: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2026-02-22T20:38:23.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3828: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2026-02-22T20:38:23.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3828: corrupt_scrub_erasure: epoch=101 2026-02-22T20:38:23.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-22T20:38:23.029 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:38:23.029 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:38:23.029 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:38:23.029 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:38:23.030 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:38:23.030 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:38:23.030 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:38:23.030 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:38:23.030 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:38:23.030 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:38:23.030 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:38:23.030 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:38:23.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: jq .inconsistents 2026-02-22T20:38:23.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:38:23.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-22T20:38:23.048 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:38:23.048 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:38:23.048 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:38:23.048 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:38:23.048 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:38:23.048 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:38:23.048 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:38:23.048 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:38:23.049 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:38:23.049 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:38:23.049 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:38:23.049 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-22T20:38:23.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: jq .inconsistents 2026-02-22T20:38:23.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:38:23.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4459: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:38:23.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:38:23.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4460: corrupt_scrub_erasure: test no = yes 2026-02-22T20:38:23.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4465: corrupt_scrub_erasure: test '' = yes 2026-02-22T20:38:23.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4470: corrupt_scrub_erasure: pg_deep_scrub 3.0 2026-02-22T20:38:23.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-22T20:38:23.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-22T20:38:23.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:38:23.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:38:23.075 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:38:23.075 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:38:23.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:38:23.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:38:23.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:38:23.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:38:23.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:38:23.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:38:23.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:38:23.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:38:23.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:38:23.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:38:23.529 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:23.529 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:38:23.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:38:23.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:23.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:38:23.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696902 2026-02-22T20:38:23.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696902 2026-02-22T20:38:23.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696902' 2026-02-22T20:38:23.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:23.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:38:23.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089354 2026-02-22T20:38:23.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089354 2026-02-22T20:38:23.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696902 1-382252089354' 2026-02-22T20:38:23.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:23.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:38:23.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743441 2026-02-22T20:38:23.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743441 2026-02-22T20:38:23.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-433791696902 1-382252089354 2-296352743441' 2026-02-22T20:38:23.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:23.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-433791696902 2026-02-22T20:38:23.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:23.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:38:23.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-433791696902 2026-02-22T20:38:23.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:23.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696902 2026-02-22T20:38:23.748 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 433791696902 2026-02-22T20:38:23.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 433791696902' 2026-02-22T20:38:23.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:23.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696901 -lt 433791696902 2026-02-22T20:38:23.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:38:24.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:38:24.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:25.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696903 -lt 433791696902 2026-02-22T20:38:25.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:25.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089354 2026-02-22T20:38:25.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:25.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:38:25.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089354 2026-02-22T20:38:25.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:25.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089354 2026-02-22T20:38:25.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089354' 2026-02-22T20:38:25.074 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 382252089354 2026-02-22T20:38:25.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:38:25.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089354 -lt 382252089354 2026-02-22T20:38:25.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:25.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743441 2026-02-22T20:38:25.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:25.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:38:25.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743441 2026-02-22T20:38:25.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:25.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743441 2026-02-22T20:38:25.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743441' 2026-02-22T20:38:25.247 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 296352743441 2026-02-22T20:38:25.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:38:25.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743441 -lt 296352743441 2026-02-22T20:38:25.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:38:25.412 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:38:25.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:38:25.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:38:25.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:38:25.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:38:25.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:38:25.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:38:25.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-22T20:38:25.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:38:25.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:38:25.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:38:25.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:38:25.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:38:25.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:38:25.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:38:25.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:38:25.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:25.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-22T20:38:25.789 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2026-02-22T20:38:25.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-22T20:37:06.108452+0000 last_deep_scrub_stamp 2026-02-22T20:38:25.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:38:25.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:25.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-22T20:38:25.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:38:25.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:38:25.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:38:25.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:38:25.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:38:25.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:38:25.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:38:25.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:37:06.108452+0000 '>' 2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:25.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:38:26.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:38:26.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:38:26.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:38:26.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:38:26.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:38:26.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:38:26.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:38:27.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:37:06.108452+0000 '>' 2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:27.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:38:28.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:38:28.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:38:28.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:38:28.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:38:28.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:38:28.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:38:28.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:38:28.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:26.212528+0000 '>' 2026-02-22T20:37:06.108452+0000 2026-02-22T20:38:28.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:38:28.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4472: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2026-02-22T20:38:28.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4474: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2026-02-22T20:38:28.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4474: corrupt_scrub_erasure: test 1 = 1 2026-02-22T20:38:28.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4476: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-22T20:38:28.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4476: corrupt_scrub_erasure: test 3.0 = 3.0 2026-02-22T20:38:28.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4478: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2026-02-22T20:38:28.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4479: corrupt_scrub_erasure: cp td/osd-scrub-repair/json /tmp/rrr6.json 2026-02-22T20:38:28.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4481: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4481: corrupt_scrub_erasure: epoch=101 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4483: corrupt_scrub_erasure: '[' false = true ']' 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5148: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:38:28.349 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:38:28.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5148: corrupt_scrub_erasure: jq .inconsistents 2026-02-22T20:38:28.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5148: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: jq .inconsistents 2026-02-22T20:38:28.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:38:28.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:38:28.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:38:28.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: corrupt_scrub_erasure: test no = yes 2026-02-22T20:38:28.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: corrupt_scrub_erasure: test '' = yes 2026-02-22T20:38:28.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2026-02-22T20:38:28.593 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' removed 2026-02-22T20:38:28.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:38:28.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:38:28.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:38:28.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:38:28.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:38:28.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:38:28.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:38:28.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:38:28.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:38:28.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:38:28.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:38:28.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:38:28.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:38:28.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:38:28.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:38:28.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:38:28.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:38:28.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:38:28.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:38:28.721 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:38:28.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:38:28.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:38:28.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:38:28.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:38:28.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:28.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:28.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:38:28.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:38:28.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:38:28.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:38:28.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:38:28.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:38:28.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:38:28.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:38:28.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:38:28.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:38:28.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:38:28.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:38:28.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:38:28.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:38:28.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:38:28.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:38:28.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:38:28.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:38:28.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:38:28.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:38:28.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:38:28.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:38:28.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:38:28.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:38:28.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:38:28.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:38:28.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:38:28.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:38:28.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:38:28.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:38:28.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:28.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:28.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:38:28.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:38:28.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:38:28.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:38:28.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:38:28.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:28.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:28.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:38:28.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:38:28.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:38:28.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:38:28.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:38:28.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_erasure_overwrites td/osd-scrub-repair 2026-02-22T20:38:28.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5916: TEST_corrupt_scrub_erasure_overwrites: '[' true = true ']' 2026-02-22T20:38:28.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5917: TEST_corrupt_scrub_erasure_overwrites: corrupt_scrub_erasure td/osd-scrub-repair true 2026-02-22T20:38:28.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3736: corrupt_scrub_erasure: local dir=td/osd-scrub-repair 2026-02-22T20:38:28.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3737: corrupt_scrub_erasure: local allow_overwrites=true 2026-02-22T20:38:28.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3738: corrupt_scrub_erasure: local poolname=ecpool 2026-02-22T20:38:28.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3739: corrupt_scrub_erasure: local total_objs=7 2026-02-22T20:38:28.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3741: corrupt_scrub_erasure: run_mon td/osd-scrub-repair a 2026-02-22T20:38:28.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:38:28.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:38:28.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:38:28.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:38:28.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:38:28.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:38:28.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:38:28.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:38:28.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:38:28.791 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:38:28.792 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:28.792 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:28.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:28.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:38:28.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:38:28.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:38:28.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:38:28.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:38:28.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:38:28.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:38:28.821 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:38:28.821 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:38:28.821 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:38:28.821 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:38:28.821 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:28.821 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:28.822 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:38:28.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:38:28.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:38:28.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:38:28.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:38:28.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:38:28.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:38:28.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:38:28.878 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:38:28.878 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:38:28.878 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:38:28.878 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:38:28.878 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:28.878 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:28.879 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:38:28.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:38:28.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:38:28.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3742: corrupt_scrub_erasure: run_mgr td/osd-scrub-repair x 2026-02-22T20:38:28.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:38:28.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:38:28.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:38:28.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:38:28.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:38:28.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:38:29.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:38:29.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:38:29.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:38:29.028 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:38:29.028 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:29.028 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:29.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:29.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:38:29.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:38:29.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: seq 0 2 2026-02-22T20:38:29.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-22T20:38:29.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 0 2026-02-22T20:38:29.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:38:29.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:38:29.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:38:29.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:38:29.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:38:29.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:38:29.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:38:29.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:38:29.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:29.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:29.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:38:29.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:38:29.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:38:29.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:38:29.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:38:29.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:38:29.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:38:29.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:38:29.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:38:29.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:38:29.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:38:29.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:38:29.057 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:38:29.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=03e639bb-21c5-4e97-9f91-796824bfaceb 2026-02-22T20:38:29.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 03e639bb-21c5-4e97-9f91-796824bfaceb' 2026-02-22T20:38:29.058 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 03e639bb-21c5-4e97-9f91-796824bfaceb 2026-02-22T20:38:29.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:38:29.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBFaZtpatxhAxAApskzN9eCMm8vQbsCfkkj2Q== 2026-02-22T20:38:29.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBFaZtpatxhAxAApskzN9eCMm8vQbsCfkkj2Q=="}' 2026-02-22T20:38:29.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 03e639bb-21c5-4e97-9f91-796824bfaceb -i td/osd-scrub-repair/0/new.json 2026-02-22T20:38:29.157 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:38:29.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:38:29.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFaZtpatxhAxAApskzN9eCMm8vQbsCfkkj2Q== --osd-uuid 03e639bb-21c5-4e97-9f91-796824bfaceb 2026-02-22T20:38:29.183 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:29.170+0000 7f4f145eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:29.184 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:29.170+0000 7f4f145eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:29.185 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:29.174+0000 7f4f145eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:29.185 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:29.174+0000 7f4f145eca40 -1 bdev(0x562108b76c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:38:29.185 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:29.174+0000 7f4f145eca40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:38:29.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:38:29.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:38:29.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:38:29.912 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:38:29.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:38:30.110 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:38:30.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:38:30.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:38:30.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:38:30.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:38:30.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:38:30.134 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:30.122+0000 7f08fe6c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:30.136 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:30.122+0000 7f08fe6c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:30.137 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:30.126+0000 7f08fe6c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:30.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:38:30.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:38:30.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:38:30.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:38:30.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:38:30.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:30.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:38:30.273 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:38:30.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:30.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:38:30.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:38:30.526 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:30.514+0000 7f08fe6c9a40 -1 Falling back to public interface 2026-02-22T20:38:31.068 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:31.054+0000 7f08fe6c9a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:38:31.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:38:31.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:31.435 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:38:31.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:38:31.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:31.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:38:31.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:38:32.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:38:32.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:32.610 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:38:32.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:38:32.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:32.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:38:32.774 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2222903467,v1:127.0.0.1:6803/2222903467] [v2:127.0.0.1:6804/2222903467,v1:127.0.0.1:6805/2222903467] exists,up 03e639bb-21c5-4e97-9f91-796824bfaceb 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 1 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:38:32.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:38:32.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:38:32.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:38:32.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:38:32.776 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:38:32.776 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:32.776 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:32.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:38:32.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:38:32.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:38:32.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3d50fe25-bdf8-41ba-b41c-60fee31f2877 2026-02-22T20:38:32.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3d50fe25-bdf8-41ba-b41c-60fee31f2877' 2026-02-22T20:38:32.779 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 3d50fe25-bdf8-41ba-b41c-60fee31f2877 2026-02-22T20:38:32.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:38:32.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBIaZtpMe2YLhAAh/j42nZwYaro+mNg5yaiUA== 2026-02-22T20:38:32.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBIaZtpMe2YLhAAh/j42nZwYaro+mNg5yaiUA=="}' 2026-02-22T20:38:32.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3d50fe25-bdf8-41ba-b41c-60fee31f2877 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:38:32.952 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:38:32.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:38:32.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIaZtpMe2YLhAAh/j42nZwYaro+mNg5yaiUA== --osd-uuid 3d50fe25-bdf8-41ba-b41c-60fee31f2877 2026-02-22T20:38:32.979 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:32.966+0000 7f19b99c3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:32.980 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:32.966+0000 7f19b99c3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:32.981 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:32.970+0000 7f19b99c3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:32.982 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:32.970+0000 7f19b99c3a40 -1 bdev(0x5625a2098c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:38:32.982 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:32.970+0000 7f19b99c3a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:38:33.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:38:33.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:38:33.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:38:33.645 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:38:33.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:38:33.844 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:38:33.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:38:33.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:38:33.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:38:33.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:38:33.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:38:33.870 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:33.858+0000 7efddcd29a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:33.872 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:33.858+0000 7efddcd29a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:33.873 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:33.862+0000 7efddcd29a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:34.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:38:34.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:38:34.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:38:34.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:38:34.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:38:34.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:34.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:38:34.005 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:38:34.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:34.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:38:34.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:38:34.338 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:34.326+0000 7efddcd29a40 -1 Falling back to public interface 2026-02-22T20:38:34.878 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:34.866+0000 7efddcd29a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:38:35.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:38:35.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:35.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:38:35.164 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:38:35.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:35.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:38:35.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:38:36.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:38:36.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:36.325 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:38:36.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:38:36.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:36.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:38:36.484 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1813312887,v1:127.0.0.1:6811/1813312887] [v2:127.0.0.1:6812/1813312887,v1:127.0.0.1:6813/1813312887] exists,up 3d50fe25-bdf8-41ba-b41c-60fee31f2877 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3743: corrupt_scrub_erasure: for id in $(seq 0 2) 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3744: corrupt_scrub_erasure: run_osd td/osd-scrub-repair 2 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:38:36.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:38:36.486 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:38:36.486 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:36.486 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:36.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:38:36.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:38:36.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:38:36.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1c294c1c-11e4-47b3-8bf6-646c7c40111c 2026-02-22T20:38:36.490 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 1c294c1c-11e4-47b3-8bf6-646c7c40111c 2026-02-22T20:38:36.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 1c294c1c-11e4-47b3-8bf6-646c7c40111c' 2026-02-22T20:38:36.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:38:36.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBMaZtp1fxhHRAAu9r5/X3JaVYBtqvTwF9S6Q== 2026-02-22T20:38:36.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBMaZtp1fxhHRAAu9r5/X3JaVYBtqvTwF9S6Q=="}' 2026-02-22T20:38:36.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1c294c1c-11e4-47b3-8bf6-646c7c40111c -i td/osd-scrub-repair/2/new.json 2026-02-22T20:38:36.665 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:38:36.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:38:36.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMaZtp1fxhHRAAu9r5/X3JaVYBtqvTwF9S6Q== --osd-uuid 1c294c1c-11e4-47b3-8bf6-646c7c40111c 2026-02-22T20:38:36.699 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:36.686+0000 7f007883fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:36.701 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:36.690+0000 7f007883fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:36.702 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:36.690+0000 7f007883fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:36.703 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:36.690+0000 7f007883fa40 -1 bdev(0x55886d262c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:38:36.703 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:36.690+0000 7f007883fa40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:38:37.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:38:37.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:38:37.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:38:37.390 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:38:37.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:38:37.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:38:37.590 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:38:37.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:38:37.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:38:37.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:38:37.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:38:37.615 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:37.602+0000 7f4e20c10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:37.617 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:37.606+0000 7f4e20c10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:37.618 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:37.606+0000 7f4e20c10a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:37.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:38:37.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:38:37.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:38:37.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:38:37.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:38:37.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:37.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:38:37.753 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:38:37.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:37.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:38:37.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:38:38.030 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:38.018+0000 7f4e20c10a40 -1 Falling back to public interface 2026-02-22T20:38:38.582 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:38.570+0000 7f4e20c10a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:38:38.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:38:38.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:38.915 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:38:38.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:38:38.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:38.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:38:39.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:38:40.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:38:40.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:40.079 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:38:40.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:38:40.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:40.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:38:40.241 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1687942488,v1:127.0.0.1:6819/1687942488] [v2:127.0.0.1:6820/1687942488,v1:127.0.0.1:6821/1687942488] exists,up 1c294c1c-11e4-47b3-8bf6-646c7c40111c 2026-02-22T20:38:40.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:38:40.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:38:40.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:38:40.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3746: corrupt_scrub_erasure: create_rbd_pool 2026-02-22T20:38:40.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:38:40.392 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:38:40.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:38:40.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:38:40.611 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:38:40.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:38:41.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:38:41.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3747: corrupt_scrub_erasure: create_pool foo 1 2026-02-22T20:38:41.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-22T20:38:42.086 INFO:tasks.workunit.client.0.trial150.stderr:pool 'foo' created 2026-02-22T20:38:42.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:38:43.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3749: corrupt_scrub_erasure: create_ec_pool ecpool true k=2 m=1 stripe_unit=2K --force 2026-02-22T20:38:43.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-22T20:38:43.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:38:43.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-22T20:38:43.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:38:43.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 stripe_unit=2K --force 2026-02-22T20:38:43.304 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:38:43.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-22T20:38:43.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-22T20:38:43.611 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' created 2026-02-22T20:38:43.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:38:44.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-22T20:38:44.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-22T20:38:44.833 INFO:tasks.workunit.client.0.trial150.stderr:set pool 3 allow_ec_overwrites to true 2026-02-22T20:38:44.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:38:44.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:38:44.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:38:44.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:38:44.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:38:44.846 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:38:44.846 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:38:44.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:38:44.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:38:44.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:38:44.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:38:44.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:38:44.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:38:44.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:38:44.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:38:44.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:38:45.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:38:45.118 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:45.118 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:38:45.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:38:45.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:45.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:38:45.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-22T20:38:45.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-22T20:38:45.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T20:38:45.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:45.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:38:45.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-22T20:38:45.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-22T20:38:45.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-22T20:38:45.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:45.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:38:45.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-22T20:38:45.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-22T20:38:45.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-38654705667 2-60129542147' 2026-02-22T20:38:45.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:45.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-22T20:38:45.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:45.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:38:45.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-22T20:38:45.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:45.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-22T20:38:45.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T20:38:45.339 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836484 2026-02-22T20:38:45.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:38:45.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:38:46.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:38:46.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:46.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-22T20:38:46.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:46.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-22T20:38:46.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:46.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:38:46.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-22T20:38:46.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:46.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-22T20:38:46.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-22T20:38:46.669 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705667 2026-02-22T20:38:46.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:38:46.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-22T20:38:46.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:46.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-22T20:38:46.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:46.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:38:46.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-22T20:38:46.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:46.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-22T20:38:46.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-22T20:38:46.837 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542147 2026-02-22T20:38:46.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:38:47.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542147 -lt 60129542147 2026-02-22T20:38:47.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:38:47.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:38:47.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:38:47.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:38:47.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:38:47.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:38:47.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:38:47.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:38:47.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:38:47.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:38:47.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:38:47.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:38:47.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:38:47.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:38:47.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:38:47.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:38:47.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:38:47.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:38:47.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:38:47.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3750: corrupt_scrub_erasure: wait_for_clean 2026-02-22T20:38:47.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:38:47.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:38:47.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:38:47.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:38:47.550 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:38:47.550 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:38:47.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:38:47.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:38:47.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:38:47.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:38:47.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:38:47.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:38:47.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:38:47.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:38:47.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:38:47.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:38:47.818 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:47.818 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:38:47.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:38:47.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:47.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:38:47.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-22T20:38:47.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-22T20:38:47.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-22T20:38:47.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:47.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:38:47.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-22T20:38:47.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-22T20:38:47.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705669' 2026-02-22T20:38:47.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:47.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:38:48.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-22T20:38:48.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-22T20:38:48.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-38654705669 2-60129542148' 2026-02-22T20:38:48.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:48.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-22T20:38:48.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:48.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:38:48.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-22T20:38:48.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:48.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-22T20:38:48.039 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836486 2026-02-22T20:38:48.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-22T20:38:48.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:48.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-22T20:38:48.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:38:49.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:38:49.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:49.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-22T20:38:49.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:38:50.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:38:50.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:50.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-22T20:38:50.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:50.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-22T20:38:50.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:50.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:38:50.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-22T20:38:50.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:50.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-22T20:38:50.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-22T20:38:50.531 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705669 2026-02-22T20:38:50.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:38:50.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-22T20:38:50.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:50.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-22T20:38:50.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:50.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:38:50.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-22T20:38:50.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:50.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-22T20:38:50.695 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542148 2026-02-22T20:38:50.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-22T20:38:50.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:38:50.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-22T20:38:50.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:38:50.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:38:50.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:38:51.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:38:51.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:38:51.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:38:51.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:38:51.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:38:51.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:38:51.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:38:51.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:38:51.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:38:51.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:38:51.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:38:51.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:38:51.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:38:51.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:38:51.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:38:51.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: seq 1 7 2026-02-22T20:38:51.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:38:51.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ1 2026-02-22T20:38:51.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ1 2026-02-22T20:38:51.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:38:51.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:38:51.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ1 2026-02-22T20:38:51.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:38:51.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:38:51.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:38:51.639 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:38:51.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:38:51.840 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:38:51.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:38:51.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:38:51.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:38:51.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 1 % 2 2026-02-22T20:38:51.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-22T20:38:51.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:38:51.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3761: corrupt_scrub_erasure: local payload=UVWXYZZZ 2026-02-22T20:38:51.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3762: corrupt_scrub_erasure: echo UVWXYZZZ 2026-02-22T20:38:51.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3763: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:38:51.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:38:51.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:38:51.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:38:51.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:38:51.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:38:51.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:38:51.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:38:51.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:38:51.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:38:51.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:38:51.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:38:51.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:38:51.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:38:51.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:38:51.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:38:52.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:38:52.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:38:52.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:38:52.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:38:52.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:38:52.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:38:52.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:38:52.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:38:52.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:38:52.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:38:52.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:38:52.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:38:52.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:38:52.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:38:52.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:38:52.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:38:52.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:38:52.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:38:52.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:38:52.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:38:52.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:38:52.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:38:52.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:38:52.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:38:52.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:38:52.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:38:52.489 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:38:52.489 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:52.489 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:52.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:52.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:52.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:38:52.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:38:52.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:38:52.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:38:52.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:38:52.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:38:52.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:38:52.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:38:52.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:38:52.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:38:52.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:38:52.491 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:38:52.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:38:52.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:38:52.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:38:52.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:38:52.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:38:52.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:38:52.508 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:52.494+0000 7ff4e00afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:52.510 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:52.498+0000 7ff4e00afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:52.510 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:52.498+0000 7ff4e00afa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:52.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:38:52.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:38:52.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:38:52.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:38:52.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:38:52.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:52.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:38:52.650 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:38:52.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:52.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:38:52.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:38:52.869 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:52.858+0000 7ff4e00afa40 -1 Falling back to public interface 2026-02-22T20:38:53.456 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:53.442+0000 7ff4e00afa40 -1 osd.1 33 log_to_monitors true 2026-02-22T20:38:53.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:38:53.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:53.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:38:53.808 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:53.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:53.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:38:53.968 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 37 up_thru 37 down_at 34 last_clean_interval [9,33) [v2:127.0.0.1:6810/21648423,v1:127.0.0.1:6811/21648423] [v2:127.0.0.1:6812/21648423,v1:127.0.0.1:6813/21648423] exists,up 3d50fe25-bdf8-41ba-b41c-60fee31f2877 2026-02-22T20:38:53.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:38:53.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:38:53.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:38:53.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:38:53.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:38:53.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:38:53.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:38:53.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:38:53.969 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:38:53.969 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:38:53.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:38:53.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:38:53.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:38:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:38:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:38:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:38:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:38:54.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:38:54.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:38:54.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:38:54.240 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:54.240 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:38:54.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:38:54.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:54.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:38:54.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-22T20:38:54.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-22T20:38:54.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-22T20:38:54.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:54.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:38:54.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789954 2026-02-22T20:38:54.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789954 2026-02-22T20:38:54.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-158913789954' 2026-02-22T20:38:54.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:54.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:38:54.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-22T20:38:54.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-22T20:38:54.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-158913789954 2-60129542150' 2026-02-22T20:38:54.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:54.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-22T20:38:54.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:54.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:38:54.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-22T20:38:54.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:54.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-22T20:38:54.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-22T20:38:54.459 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836488 2026-02-22T20:38:54.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:54.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-22T20:38:54.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:38:55.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:38:55.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:55.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-22T20:38:55.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:55.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-158913789954 2026-02-22T20:38:55.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:55.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:38:55.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-158913789954 2026-02-22T20:38:55.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:55.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789954 2026-02-22T20:38:55.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 158913789954' 2026-02-22T20:38:55.795 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 158913789954 2026-02-22T20:38:55.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:38:55.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789954 -lt 158913789954 2026-02-22T20:38:55.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:55.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-22T20:38:55.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:55.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:38:55.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-22T20:38:55.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:55.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-22T20:38:55.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-22T20:38:55.968 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542150 2026-02-22T20:38:55.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:38:56.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542151 -lt 60129542150 2026-02-22T20:38:56.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:38:56.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:38:56.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:38:56.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:38:56.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:38:56.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:38:56.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:38:56.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:38:56.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:38:56.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:38:56.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:38:56.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:38:56.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:38:56.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:38:56.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:38:56.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:38:56.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:38:56.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:38:56.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:38:56.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ2 2026-02-22T20:38:56.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ2 2026-02-22T20:38:56.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:38:56.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:38:56.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ2 2026-02-22T20:38:56.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:38:56.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:38:56.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:38:56.889 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:38:56.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:38:57.093 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:38:57.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:38:57.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:38:57.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:38:57.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 2 % 2 2026-02-22T20:38:57.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-22T20:38:57.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:38:57.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3768: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=1 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:2048 bytes (2.0 kB, 2.0 KiB) copied, 8.8236e-05 s, 23.2 MB/s 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3769: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:38:57.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:38:57.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:38:57.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:38:57.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:38:57.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:38:57.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:38:57.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:38:57.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:38:57.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:38:57.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:38:57.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:38:57.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:38:57.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:38:57.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:38:57.754 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:38:57.754 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:38:57.754 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:38:57.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:57.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:38:57.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:38:57.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:38:57.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:38:57.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:38:57.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:38:57.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:38:57.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:38:57.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:38:57.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:38:57.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:38:57.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:38:57.755 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:38:57.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:38:57.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:38:57.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:38:57.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:38:57.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:38:57.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:38:57.775 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:57.762+0000 7f0481d00a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:57.777 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:57.766+0000 7f0481d00a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:57.778 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:57.766+0000 7f0481d00a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:38:57.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:38:57.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:38:57.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:38:57.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:38:57.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:38:57.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:57.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:38:57.921 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:38:57.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:57.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:38:58.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:38:58.190 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:58.178+0000 7f0481d00a40 -1 Falling back to public interface 2026-02-22T20:38:58.818 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:38:58.806+0000 7f0481d00a40 -1 osd.0 40 log_to_monitors true 2026-02-22T20:38:59.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:38:59.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:38:59.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:38:59.080 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:59.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:38:59.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:38:59.239 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 44 up_thru 44 down_at 41 last_clean_interval [5,40) [v2:127.0.0.1:6802/1707839971,v1:127.0.0.1:6803/1707839971] [v2:127.0.0.1:6804/1707839971,v1:127.0.0.1:6805/1707839971] exists,up 03e639bb-21c5-4e97-9f91-796824bfaceb 2026-02-22T20:38:59.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:38:59.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:38:59.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:38:59.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:38:59.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:38:59.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:38:59.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:38:59.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:38:59.241 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:38:59.241 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:38:59.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:38:59.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:38:59.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:38:59.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:38:59.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:38:59.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:38:59.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:38:59.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:38:59.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:38:59.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:38:59.515 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:38:59.515 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:38:59.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:38:59.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:59.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:38:59.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561026 2026-02-22T20:38:59.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561026 2026-02-22T20:38:59.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-188978561026' 2026-02-22T20:38:59.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:59.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:38:59.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=158913789956 2026-02-22T20:38:59.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 158913789956 2026-02-22T20:38:59.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-188978561026 1-158913789956' 2026-02-22T20:38:59.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:38:59.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:38:59.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542152 2026-02-22T20:38:59.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542152 2026-02-22T20:38:59.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-188978561026 1-158913789956 2-60129542152' 2026-02-22T20:38:59.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:38:59.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-188978561026 2026-02-22T20:38:59.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:38:59.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:38:59.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-188978561026 2026-02-22T20:38:59.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:38:59.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=188978561026 2026-02-22T20:38:59.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 188978561026' 2026-02-22T20:38:59.744 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 188978561026 2026-02-22T20:38:59.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:38:59.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 188978561026 2026-02-22T20:38:59.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:00.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:39:00.916 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:01.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 188978561026 2026-02-22T20:39:01.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:02.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:39:02.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:02.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 188978561026 -lt 188978561026 2026-02-22T20:39:02.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:02.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-158913789956 2026-02-22T20:39:02.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:02.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:39:02.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-158913789956 2026-02-22T20:39:02.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:02.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=158913789956 2026-02-22T20:39:02.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 158913789956' 2026-02-22T20:39:02.242 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 158913789956 2026-02-22T20:39:02.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:02.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 158913789956 -lt 158913789956 2026-02-22T20:39:02.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:02.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542152 2026-02-22T20:39:02.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:02.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:39:02.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542152 2026-02-22T20:39:02.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:02.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542152 2026-02-22T20:39:02.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2026-02-22T20:39:02.419 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542152 2026-02-22T20:39:02.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:39:02.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542153 -lt 60129542152 2026-02-22T20:39:02.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:39:02.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:02.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:02.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:39:02.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:39:02.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:39:02.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:39:02.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:39:02.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:39:02.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:39:02.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:39:02.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:39:02.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:39:02.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:02.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:03.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:39:03.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:39:03.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:39:03.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:39:03.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ3 2026-02-22T20:39:03.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ3 2026-02-22T20:39:03.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:39:03.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:39:03.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ3 2026-02-22T20:39:03.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:39:03.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:39:03.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:39:03.306 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:39:03.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:39:03.510 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:39:03.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:39:03.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:39:03.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ3 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:39:03.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 3 % 2 2026-02-22T20:39:03.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-22T20:39:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:39:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3774: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ3 remove 2026-02-22T20:39:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:39:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:39:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:39:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:39:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ3 remove 2026-02-22T20:39:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:39:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:39:03.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:39:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:39:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:39:03.548 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:39:03.548 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:39:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:39:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:39:03.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:39:03.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:39:03.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ3 remove 2026-02-22T20:39:03.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:39:03.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:39:03.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:39:03.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:39:03.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:39:03.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ3 remove 2026-02-22T20:39:04.045 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#3:b197b25d:::EOBJ3:head# 2026-02-22T20:39:04.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:39:04.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:39:04.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:39:04.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:39:04.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:39:04.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:39:04.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:39:04.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:39:04.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:39:04.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:39:04.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:39:04.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:39:04.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:39:04.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:39:04.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:39:04.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:39:04.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:39:04.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:39:04.140 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:39:04.140 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:39:04.140 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:39:04.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:04.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:04.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:39:04.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:39:04.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:39:04.157 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:04.146+0000 7f136198aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:04.158 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:04.146+0000 7f136198aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:04.159 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:04.146+0000 7f136198aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:04.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:39:04.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:39:04.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:39:04.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:39:04.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:39:04.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:04.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:39:04.299 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:39:04.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:04.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:39:04.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:39:04.578 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:04.566+0000 7f136198aa40 -1 Falling back to public interface 2026-02-22T20:39:05.123 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:05.110+0000 7f136198aa40 -1 osd.1 48 log_to_monitors true 2026-02-22T20:39:05.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:39:05.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:05.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:39:05.464 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:05.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:05.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:39:05.624 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 52 up_thru 52 down_at 49 last_clean_interval [37,48) [v2:127.0.0.1:6810/1023991117,v1:127.0.0.1:6811/1023991117] [v2:127.0.0.1:6812/1023991117,v1:127.0.0.1:6813/1023991117] exists,up 3d50fe25-bdf8-41ba-b41c-60fee31f2877 2026-02-22T20:39:05.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:39:05.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:39:05.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:39:05.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:39:05.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:39:05.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:39:05.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:39:05.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:39:05.626 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:39:05.626 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:39:05.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:39:05.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:39:05.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:39:05.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:39:05.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:39:05.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:39:05.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:39:05.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:39:05.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:39:05.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:39:05.897 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:05.897 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:39:05.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:39:05.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:05.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:39:05.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=188978561028 2026-02-22T20:39:05.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 188978561028 2026-02-22T20:39:05.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-188978561028' 2026-02-22T20:39:05.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:05.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:39:06.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=223338299394 2026-02-22T20:39:06.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 223338299394 2026-02-22T20:39:06.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-188978561028 1-223338299394' 2026-02-22T20:39:06.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:06.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:39:06.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542155 2026-02-22T20:39:06.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542155 2026-02-22T20:39:06.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-188978561028 1-223338299394 2-60129542155' 2026-02-22T20:39:06.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:06.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-188978561028 2026-02-22T20:39:06.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:06.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:39:06.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-188978561028 2026-02-22T20:39:06.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:06.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=188978561028 2026-02-22T20:39:06.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 188978561028' 2026-02-22T20:39:06.126 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 188978561028 2026-02-22T20:39:06.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:06.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 188978561027 -lt 188978561028 2026-02-22T20:39:06.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:07.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:39:07.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:07.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 188978561027 -lt 188978561028 2026-02-22T20:39:07.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:08.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:39:08.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:08.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 188978561028 -lt 188978561028 2026-02-22T20:39:08.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:08.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-223338299394 2026-02-22T20:39:08.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:08.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:39:08.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-223338299394 2026-02-22T20:39:08.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:08.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=223338299394 2026-02-22T20:39:08.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 223338299394' 2026-02-22T20:39:08.621 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 223338299394 2026-02-22T20:39:08.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:08.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 223338299394 -lt 223338299394 2026-02-22T20:39:08.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:08.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542155 2026-02-22T20:39:08.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:08.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:39:08.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542155 2026-02-22T20:39:08.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:08.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542155 2026-02-22T20:39:08.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-22T20:39:08.792 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542155 2026-02-22T20:39:08.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:39:08.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-22T20:39:08.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:39:08.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:08.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:09.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:39:09.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:39:09.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:39:09.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:39:09.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:39:09.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:39:09.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:39:09.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:39:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:39:09.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:39:09.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:09.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:09.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:39:09.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:39:09.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:39:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:39:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ4 2026-02-22T20:39:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ4 2026-02-22T20:39:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:39:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:39:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ4 2026-02-22T20:39:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:39:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:39:09.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:39:09.682 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:39:09.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:39:09.885 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:39:09.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:39:09.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:39:09.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ4 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:39:09.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 4 % 2 2026-02-22T20:39:09.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-22T20:39:09.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:39:09.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3778: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key1-EOBJ4 val1-EOBJ4 2026-02-22T20:39:09.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3779: corrupt_scrub_erasure: rados --pool ecpool setxattr EOBJ4 key2-EOBJ4 val2-EOBJ4 2026-02-22T20:39:09.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3782: corrupt_scrub_erasure: echo -n bad-val 2026-02-22T20:39:09.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3783: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-22T20:39:09.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:39:09.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:39:09.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:39:09.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:39:09.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-22T20:39:09.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:39:09.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:39:09.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:39:09.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:39:09.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:39:09.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:39:09.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:39:09.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:39:09.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:39:09.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:39:10.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:39:10.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-22T20:39:10.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:39:10.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:39:10.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:39:10.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:39:10.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:39:10.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ4 set-attr _key1-EOBJ4 td/osd-scrub-repair/bad-val 2026-02-22T20:39:10.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:39:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:39:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:39:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:39:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:39:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:39:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:39:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:39:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:39:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:39:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:39:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:39:10.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:39:10.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:39:10.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:39:10.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:39:10.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:39:10.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:39:10.532 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:39:10.532 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:39:10.532 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:39:10.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:10.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:10.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:39:10.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:39:10.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:39:10.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:39:10.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:39:10.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:39:10.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:39:10.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:39:10.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:39:10.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:39:10.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:39:10.536 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:39:10.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:39:10.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:39:10.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:39:10.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:39:10.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:39:10.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:39:10.547 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:10.534+0000 7f03a0120a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:10.548 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:10.538+0000 7f03a0120a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:10.549 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:10.538+0000 7f03a0120a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:10.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:39:10.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:39:10.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:39:10.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:39:10.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:39:10.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:10.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:39:10.695 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:39:10.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:10.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:39:10.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:39:10.938 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:10.926+0000 7f03a0120a40 -1 Falling back to public interface 2026-02-22T20:39:11.545 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:11.534+0000 7f03a0120a40 -1 osd.0 55 log_to_monitors true 2026-02-22T20:39:11.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:39:11.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:11.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:39:11.851 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:11.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:11.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:39:12.006 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 59 up_thru 59 down_at 56 last_clean_interval [44,55) [v2:127.0.0.1:6802/3349262063,v1:127.0.0.1:6803/3349262063] [v2:127.0.0.1:6804/3349262063,v1:127.0.0.1:6805/3349262063] exists,up 03e639bb-21c5-4e97-9f91-796824bfaceb 2026-02-22T20:39:12.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:39:12.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:39:12.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:39:12.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:39:12.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:39:12.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:39:12.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:39:12.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:39:12.007 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:39:12.008 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:39:12.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:39:12.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:39:12.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:39:12.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:39:12.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:39:12.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:39:12.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:39:12.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:39:12.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:39:12.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:39:12.281 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:12.281 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:39:12.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:39:12.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:12.282 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:39:12.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=253403070466 2026-02-22T20:39:12.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 253403070466 2026-02-22T20:39:12.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-253403070466' 2026-02-22T20:39:12.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:12.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:39:12.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=223338299396 2026-02-22T20:39:12.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 223338299396 2026-02-22T20:39:12.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-253403070466 1-223338299396' 2026-02-22T20:39:12.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:12.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:39:12.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542157 2026-02-22T20:39:12.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542157 2026-02-22T20:39:12.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-253403070466 1-223338299396 2-60129542157' 2026-02-22T20:39:12.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:12.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-253403070466 2026-02-22T20:39:12.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:12.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:39:12.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-253403070466 2026-02-22T20:39:12.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:12.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=253403070466 2026-02-22T20:39:12.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 253403070466' 2026-02-22T20:39:12.504 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 253403070466 2026-02-22T20:39:12.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:12.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 188978561029 -lt 253403070466 2026-02-22T20:39:12.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:13.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:39:13.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:13.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 253403070466 -lt 253403070466 2026-02-22T20:39:13.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:13.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-223338299396 2026-02-22T20:39:13.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:13.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:39:13.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-223338299396 2026-02-22T20:39:13.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:13.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=223338299396 2026-02-22T20:39:13.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 223338299396' 2026-02-22T20:39:13.840 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 223338299396 2026-02-22T20:39:13.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:14.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 223338299396 -lt 223338299396 2026-02-22T20:39:14.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:14.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542157 2026-02-22T20:39:14.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:14.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:39:14.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542157 2026-02-22T20:39:14.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:14.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542157 2026-02-22T20:39:14.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2026-02-22T20:39:14.014 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 60129542157 2026-02-22T20:39:14.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:39:14.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542157 -lt 60129542157 2026-02-22T20:39:14.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:39:14.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:14.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:14.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:39:14.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:39:14.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:39:14.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:39:14.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:39:14.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:39:14.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:39:14.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:39:14.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:39:14.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:39:14.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:14.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3784: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:39:14.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:39:15.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:39:15.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-22T20:39:15.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:39:15.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:39:15.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T20:39:15.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:39:15.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:39:15.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 rm-attr _key2-EOBJ4 2026-02-22T20:39:15.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-22T20:39:15.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:39:15.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:39:15.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:39:15.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:39:15.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:39:15.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:39:15.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:39:15.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:39:15.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:39:15.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:39:15.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:39:15.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:39:15.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:39:15.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:39:15.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:39:15.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:39:15.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:39:15.324 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:39:15.324 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:39:15.324 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:39:15.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:15.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:15.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:39:15.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:39:15.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:39:15.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:39:15.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:39:15.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:39:15.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:39:15.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:39:15.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:39:15.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:39:15.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:39:15.327 INFO:tasks.workunit.client.0.trial150.stderr:start osd.2 2026-02-22T20:39:15.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:39:15.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-22T20:39:15.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:39:15.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:39:15.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:39:15.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:39:15.341 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:15.330+0000 7f8544940a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:15.342 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:15.330+0000 7f8544940a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:15.343 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:15.330+0000 7f8544940a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:15.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:39:15.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:39:15.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:39:15.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:39:15.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:39:15.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:15.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:39:15.485 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:39:15.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:15.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:39:15.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:39:15.806 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:15.794+0000 7f8544940a40 -1 Falling back to public interface 2026-02-22T20:39:16.431 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:16.418+0000 7f8544940a40 -1 osd.2 61 log_to_monitors true 2026-02-22T20:39:16.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:39:16.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:16.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:39:16.649 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:16.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:16.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:39:16.805 INFO:tasks.workunit.client.0.trial150.stderr:osd.2 up in weight 1 up_from 64 up_thru 64 down_at 61 last_clean_interval [14,61) [v2:127.0.0.1:6818/4157122140,v1:127.0.0.1:6819/4157122140] [v2:127.0.0.1:6820/4157122140,v1:127.0.0.1:6821/4157122140] exists,up 1c294c1c-11e4-47b3-8bf6-646c7c40111c 2026-02-22T20:39:16.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:39:16.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:39:16.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:39:16.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:39:16.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:39:16.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:39:16.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:39:16.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:39:16.806 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:39:16.806 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:39:16.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:39:16.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:39:16.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:39:16.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:39:16.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:39:16.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:39:16.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:39:16.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:39:16.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:39:17.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:39:17.084 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:17.084 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:39:17.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:39:17.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:17.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:39:17.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=253403070468 2026-02-22T20:39:17.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 253403070468 2026-02-22T20:39:17.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-253403070468' 2026-02-22T20:39:17.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:17.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:39:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=223338299398 2026-02-22T20:39:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 223338299398 2026-02-22T20:39:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-253403070468 1-223338299398' 2026-02-22T20:39:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:17.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:39:17.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=274877906946 2026-02-22T20:39:17.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 274877906946 2026-02-22T20:39:17.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-253403070468 1-223338299398 2-274877906946' 2026-02-22T20:39:17.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:17.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-253403070468 2026-02-22T20:39:17.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:17.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:39:17.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-253403070468 2026-02-22T20:39:17.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:17.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=253403070468 2026-02-22T20:39:17.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 253403070468' 2026-02-22T20:39:17.308 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 253403070468 2026-02-22T20:39:17.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:17.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 253403070466 -lt 253403070468 2026-02-22T20:39:17.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:18.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:39:18.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:18.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 253403070468 -lt 253403070468 2026-02-22T20:39:18.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:18.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-223338299398 2026-02-22T20:39:18.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:18.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:39:18.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-223338299398 2026-02-22T20:39:18.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:18.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=223338299398 2026-02-22T20:39:18.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 223338299398' 2026-02-22T20:39:18.650 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 223338299398 2026-02-22T20:39:18.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:18.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 223338299398 -lt 223338299398 2026-02-22T20:39:18.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:18.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-274877906946 2026-02-22T20:39:18.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:18.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:39:18.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-274877906946 2026-02-22T20:39:18.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:18.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=274877906946 2026-02-22T20:39:18.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 274877906946' 2026-02-22T20:39:18.821 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 274877906946 2026-02-22T20:39:18.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:39:18.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 274877906946 -lt 274877906946 2026-02-22T20:39:18.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:39:18.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:18.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:19.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:39:19.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:39:19.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:39:19.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:39:19.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:39:19.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:39:19.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:39:19.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:39:19.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:39:19.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:39:19.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:19.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:19.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:39:19.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:39:19.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:39:19.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3785: corrupt_scrub_erasure: echo -n val3-EOBJ4 2026-02-22T20:39:19.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3786: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-22T20:39:19.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:39:19.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:39:19.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T20:39:19.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:39:19.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-22T20:39:19.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:39:19.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:39:19.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T20:39:19.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:39:19.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-22T20:39:19.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:39:19.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:39:19.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:39:19.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:39:19.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:39:19.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:39:19.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-22T20:39:19.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:39:19.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:39:19.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T20:39:19.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:39:19.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:39:19.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 EOBJ4 set-attr _key3-EOBJ4 td/osd-scrub-repair/newval 2026-02-22T20:39:20.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:39:20.137 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:39:20.138 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:39:20.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:20.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:20.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:39:20.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:39:20.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:39:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:39:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:39:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:39:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:39:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:39:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:39:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:39:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:39:20.139 INFO:tasks.workunit.client.0.trial150.stderr:start osd.2 2026-02-22T20:39:20.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:39:20.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-22T20:39:20.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:39:20.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:39:20.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:39:20.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:39:20.158 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:20.146+0000 7f406dee5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:20.159 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:20.146+0000 7f406dee5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:20.160 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:20.150+0000 7f406dee5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:20.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:39:20.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:39:20.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:39:20.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:39:20.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:39:20.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:20.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:39:20.302 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:39:20.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:20.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:39:20.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:39:20.534 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:20.522+0000 7f406dee5a40 -1 Falling back to public interface 2026-02-22T20:39:21.087 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:21.074+0000 7f406dee5a40 -1 osd.2 65 log_to_monitors true 2026-02-22T20:39:21.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:39:21.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:21.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:39:21.460 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:21.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:21.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:39:21.615 INFO:tasks.workunit.client.0.trial150.stderr:osd.2 up in weight 1 up_from 69 up_thru 69 down_at 66 last_clean_interval [64,65) [v2:127.0.0.1:6818/1513266170,v1:127.0.0.1:6819/1513266170] [v2:127.0.0.1:6820/1513266170,v1:127.0.0.1:6821/1513266170] exists,up 1c294c1c-11e4-47b3-8bf6-646c7c40111c 2026-02-22T20:39:21.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:39:21.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:39:21.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:39:21.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:39:21.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:39:21.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:39:21.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:39:21.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:39:21.617 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:39:21.617 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:39:21.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:39:21.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:39:21.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:39:21.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:39:21.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:39:21.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:39:21.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:39:21.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:39:21.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:39:21.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:39:21.889 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:21.889 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:39:21.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:39:21.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:21.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:39:21.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=253403070470 2026-02-22T20:39:21.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 253403070470 2026-02-22T20:39:21.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-253403070470' 2026-02-22T20:39:21.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:21.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:39:22.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=223338299400 2026-02-22T20:39:22.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 223338299400 2026-02-22T20:39:22.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-253403070470 1-223338299400' 2026-02-22T20:39:22.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:22.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:39:22.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743426 2026-02-22T20:39:22.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743426 2026-02-22T20:39:22.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-253403070470 1-223338299400 2-296352743426' 2026-02-22T20:39:22.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:22.109 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-253403070470 2026-02-22T20:39:22.109 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:22.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:39:22.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-253403070470 2026-02-22T20:39:22.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:22.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=253403070470 2026-02-22T20:39:22.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 253403070470' 2026-02-22T20:39:22.113 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 253403070470 2026-02-22T20:39:22.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:22.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 253403070468 -lt 253403070470 2026-02-22T20:39:22.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:23.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:39:23.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:23.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 253403070468 -lt 253403070470 2026-02-22T20:39:23.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:24.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:39:24.434 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:24.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 253403070470 -lt 253403070470 2026-02-22T20:39:24.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:24.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-223338299400 2026-02-22T20:39:24.597 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:24.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:39:24.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-223338299400 2026-02-22T20:39:24.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:24.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=223338299400 2026-02-22T20:39:24.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 223338299400' 2026-02-22T20:39:24.600 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 223338299400 2026-02-22T20:39:24.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:24.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 223338299400 -lt 223338299400 2026-02-22T20:39:24.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:24.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743426 2026-02-22T20:39:24.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:24.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:39:24.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743426 2026-02-22T20:39:24.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:24.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743426 2026-02-22T20:39:24.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743426' 2026-02-22T20:39:24.775 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 296352743426 2026-02-22T20:39:24.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:39:24.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743426 -lt 296352743426 2026-02-22T20:39:24.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:39:24.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:24.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:25.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:39:25.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:39:25.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:39:25.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:39:25.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:39:25.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:39:25.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:39:25.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:39:25.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:39:25.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:39:25.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:25.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:25.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:39:25.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:39:25.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:39:25.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3787: corrupt_scrub_erasure: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2026-02-22T20:39:25.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:39:25.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ5 2026-02-22T20:39:25.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ5 2026-02-22T20:39:25.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:39:25.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:39:25.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ5 2026-02-22T20:39:25.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:39:25.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:39:25.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:39:25.725 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:39:25.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:39:25.928 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:39:25.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:39:25.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:39:25.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ5 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:39:25.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 5 % 2 2026-02-22T20:39:25.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-22T20:39:25.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:39:25.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3792: corrupt_scrub_erasure: dd if=/dev/urandom of=td/osd-scrub-repair/CORRUPT bs=2048 count=2 2026-02-22T20:39:25.964 INFO:tasks.workunit.client.0.trial150.stderr:2+0 records in 2026-02-22T20:39:25.964 INFO:tasks.workunit.client.0.trial150.stderr:2+0 records out 2026-02-22T20:39:25.964 INFO:tasks.workunit.client.0.trial150.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 5.7428e-05 s, 71.3 MB/s 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3793: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:39:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:39:26.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:39:26.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:39:26.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:39:26.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:39:26.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:39:26.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:39:26.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:39:26.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ5 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:39:26.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:39:26.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:39:26.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:39:26.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:39:26.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:39:26.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:39:26.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:39:26.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:39:26.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:39:26.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:39:26.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:39:26.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:39:26.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:39:26.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:39:26.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:39:26.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:39:26.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:39:26.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:39:26.555 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:39:26.555 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:39:26.555 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:39:26.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:26.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:26.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:39:26.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:39:26.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:39:26.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:39:26.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:39:26.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:39:26.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:39:26.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:39:26.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:39:26.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:39:26.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:39:26.557 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:39:26.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:39:26.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:39:26.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:39:26.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:39:26.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:39:26.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:39:26.575 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:26.562+0000 7f4f25854a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:26.577 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:26.566+0000 7f4f25854a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:26.577 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:26.566+0000 7f4f25854a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:26.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:39:26.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:39:26.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:39:26.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:39:26.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:39:26.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:26.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:39:26.716 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:39:26.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:26.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:39:26.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:39:26.958 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:26.946+0000 7f4f25854a40 -1 Falling back to public interface 2026-02-22T20:39:27.620 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:27.606+0000 7f4f25854a40 -1 osd.1 73 log_to_monitors true 2026-02-22T20:39:27.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:39:27.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:27.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:39:27.877 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:27.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:27.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:39:28.032 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [52,73) [v2:127.0.0.1:6810/232833294,v1:127.0.0.1:6811/232833294] [v2:127.0.0.1:6812/232833294,v1:127.0.0.1:6813/232833294] exists,up 3d50fe25-bdf8-41ba-b41c-60fee31f2877 2026-02-22T20:39:28.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:39:28.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:39:28.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:39:28.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:39:28.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:39:28.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:39:28.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:39:28.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:39:28.034 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:39:28.034 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:39:28.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:39:28.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:39:28.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:39:28.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:39:28.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:39:28.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:39:28.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:39:28.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:39:28.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:39:28.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:39:28.302 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:28.302 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:39:28.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:39:28.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:28.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:39:28.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=253403070472 2026-02-22T20:39:28.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 253403070472 2026-02-22T20:39:28.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-253403070472' 2026-02-22T20:39:28.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:28.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:39:28.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481794 2026-02-22T20:39:28.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481794 2026-02-22T20:39:28.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-253403070472 1-330712481794' 2026-02-22T20:39:28.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:28.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:39:28.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743428 2026-02-22T20:39:28.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743428 2026-02-22T20:39:28.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-253403070472 1-330712481794 2-296352743428' 2026-02-22T20:39:28.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:28.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-253403070472 2026-02-22T20:39:28.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:28.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:39:28.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-253403070472 2026-02-22T20:39:28.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:28.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=253403070472 2026-02-22T20:39:28.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 253403070472' 2026-02-22T20:39:28.527 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 253403070472 2026-02-22T20:39:28.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:28.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 253403070471 -lt 253403070472 2026-02-22T20:39:28.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:29.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:39:29.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:29.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 253403070472 -lt 253403070472 2026-02-22T20:39:29.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:29.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-330712481794 2026-02-22T20:39:29.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:29.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:39:29.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-330712481794 2026-02-22T20:39:29.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:29.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=330712481794 2026-02-22T20:39:29.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2026-02-22T20:39:29.859 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 330712481794 2026-02-22T20:39:29.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:30.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481794 -lt 330712481794 2026-02-22T20:39:30.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:30.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743428 2026-02-22T20:39:30.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:30.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:39:30.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743428 2026-02-22T20:39:30.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:30.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743428 2026-02-22T20:39:30.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743428' 2026-02-22T20:39:30.032 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 296352743428 2026-02-22T20:39:30.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:39:30.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743428 -lt 296352743428 2026-02-22T20:39:30.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:39:30.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:30.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:30.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:39:30.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:39:30.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:39:30.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:39:30.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:39:30.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:39:30.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:39:30.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:39:30.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:39:30.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:39:30.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:30.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:30.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:39:30.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:39:30.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:39:30.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:39:30.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ6 2026-02-22T20:39:30.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ6 2026-02-22T20:39:30.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:39:30.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:39:30.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ6 2026-02-22T20:39:30.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:39:30.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:39:30.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:39:30.944 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:39:30.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:39:31.148 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:39:31.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:39:31.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:39:31.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ6 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:39:31.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 6 % 2 2026-02-22T20:39:31.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=0 2026-02-22T20:39:31.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:39:31.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3797: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-22T20:39:31.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:39:31.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:39:31.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:39:31.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:39:31.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-22T20:39:31.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:39:31.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:39:31.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:39:31.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:39:31.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:39:31.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:39:31.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:39:31.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:39:31.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:39:31.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:39:31.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:39:31.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ6 rm-attr hinfo_key 2026-02-22T20:39:31.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:39:31.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:39:31.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:39:31.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:39:31.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:39:31.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ6 rm-attr hinfo_key 2026-02-22T20:39:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:39:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:39:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:39:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:39:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:39:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:39:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:39:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:39:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:39:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:39:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:39:31.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:39:31.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:39:31.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:39:31.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:39:31.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:39:31.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:39:31.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:39:31.786 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:39:31.786 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:39:31.786 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:39:31.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:39:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:39:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:39:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:39:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:39:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:39:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:39:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:39:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:39:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:39:31.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:39:31.788 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:39:31.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:39:31.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:39:31.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:39:31.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:39:31.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:39:31.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:39:31.807 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:31.794+0000 7ff232616a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:31.809 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:31.798+0000 7ff232616a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:31.810 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:31.798+0000 7ff232616a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:31.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:39:31.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:39:31.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:39:31.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:39:31.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:39:31.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:31.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:39:31.957 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:39:31.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:31.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:39:32.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:39:32.142 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:32.130+0000 7ff232616a40 -1 Falling back to public interface 2026-02-22T20:39:32.747 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:32.734+0000 7ff232616a40 -1 osd.0 80 log_to_monitors true 2026-02-22T20:39:33.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:39:33.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:33.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:39:33.113 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:33.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:33.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:39:33.267 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [59,80) [v2:127.0.0.1:6802/4116463449,v1:127.0.0.1:6803/4116463449] [v2:127.0.0.1:6804/4116463449,v1:127.0.0.1:6805/4116463449] exists,up 03e639bb-21c5-4e97-9f91-796824bfaceb 2026-02-22T20:39:33.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:39:33.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:39:33.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:39:33.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:39:33.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:39:33.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:39:33.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:39:33.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:39:33.269 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:39:33.269 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:39:33.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:39:33.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:39:33.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:39:33.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:39:33.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:39:33.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:39:33.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:39:33.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:39:33.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:39:33.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:39:33.539 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:33.539 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:39:33.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:39:33.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:33.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:39:33.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=360777252866 2026-02-22T20:39:33.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 360777252866 2026-02-22T20:39:33.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866' 2026-02-22T20:39:33.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:33.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:39:33.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481796 2026-02-22T20:39:33.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481796 2026-02-22T20:39:33.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866 1-330712481796' 2026-02-22T20:39:33.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:33.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:39:33.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743430 2026-02-22T20:39:33.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743430 2026-02-22T20:39:33.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866 1-330712481796 2-296352743430' 2026-02-22T20:39:33.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:33.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-360777252866 2026-02-22T20:39:33.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:33.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:39:33.758 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-360777252866 2026-02-22T20:39:33.758 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:33.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=360777252866 2026-02-22T20:39:33.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2026-02-22T20:39:33.760 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 360777252866 2026-02-22T20:39:33.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:33.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 360777252866 2026-02-22T20:39:33.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:34.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:39:34.916 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:35.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 360777252866 2026-02-22T20:39:35.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:36.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:39:36.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:36.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252866 -lt 360777252866 2026-02-22T20:39:36.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:36.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-330712481796 2026-02-22T20:39:36.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:36.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:39:36.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-330712481796 2026-02-22T20:39:36.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:36.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=330712481796 2026-02-22T20:39:36.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 330712481796' 2026-02-22T20:39:36.240 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 330712481796 2026-02-22T20:39:36.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:36.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481796 -lt 330712481796 2026-02-22T20:39:36.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:36.395 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743430 2026-02-22T20:39:36.395 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:36.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:39:36.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743430 2026-02-22T20:39:36.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:36.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743430 2026-02-22T20:39:36.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743430' 2026-02-22T20:39:36.399 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 296352743430 2026-02-22T20:39:36.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:39:36.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743430 -lt 296352743430 2026-02-22T20:39:36.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:39:36.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:36.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:36.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:39:36.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:39:36.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:39:36.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:39:36.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:39:36.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:39:36.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:39:36.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:39:36.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:39:36.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:39:36.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:36.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:37.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3798: corrupt_scrub_erasure: echo -n bad-val 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3799: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:39:37.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:39:37.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:39:37.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-22T20:39:37.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:39:37.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:39:37.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:39:37.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:39:37.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:39:37.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 EOBJ6 set-attr hinfo_key td/osd-scrub-repair/bad-val 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:39:37.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:39:37.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:39:37.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:39:37.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:39:37.698 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:39:37.698 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:39:37.698 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:39:37.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:39:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:39:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:39:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:39:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:39:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:39:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:39:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:39:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:39:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:39:37.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:39:37.700 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:39:37.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:39:37.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:39:37.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:39:37.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:39:37.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:39:37.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:39:37.717 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:37.706+0000 7fb1e753da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:37.719 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:37.706+0000 7fb1e753da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:37.719 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:37.706+0000 7fb1e753da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:37.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:39:37.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:39:37.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:39:37.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:39:37.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:39:37.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:37.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:39:37.861 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:39:37.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:37.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:39:38.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:39:38.094 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:38.082+0000 7fb1e753da40 -1 Falling back to public interface 2026-02-22T20:39:38.667 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:38.654+0000 7fb1e753da40 -1 osd.1 85 log_to_monitors true 2026-02-22T20:39:39.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:39:39.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:39.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:39:39.021 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:39.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:39.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:39:39.176 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 86 last_clean_interval [77,85) [v2:127.0.0.1:6810/3591769549,v1:127.0.0.1:6811/3591769549] [v2:127.0.0.1:6812/3591769549,v1:127.0.0.1:6813/3591769549] exists,up 3d50fe25-bdf8-41ba-b41c-60fee31f2877 2026-02-22T20:39:39.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:39:39.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:39:39.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:39:39.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:39:39.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:39:39.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:39:39.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:39:39.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:39:39.177 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:39:39.177 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:39:39.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:39:39.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:39:39.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:39:39.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:39:39.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:39:39.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:39:39.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:39:39.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:39:39.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:39:39.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:39:39.444 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:39.444 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:39:39.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:39:39.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:39.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:39:39.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=360777252868 2026-02-22T20:39:39.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 360777252868 2026-02-22T20:39:39.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252868' 2026-02-22T20:39:39.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:39.520 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:39:39.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089346 2026-02-22T20:39:39.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089346 2026-02-22T20:39:39.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252868 1-382252089346' 2026-02-22T20:39:39.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:39.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:39:39.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743432 2026-02-22T20:39:39.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743432 2026-02-22T20:39:39.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252868 1-382252089346 2-296352743432' 2026-02-22T20:39:39.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:39.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-360777252868 2026-02-22T20:39:39.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:39.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:39:39.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-360777252868 2026-02-22T20:39:39.666 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:39.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=360777252868 2026-02-22T20:39:39.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 360777252868' 2026-02-22T20:39:39.667 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 360777252868 2026-02-22T20:39:39.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:39.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252868 -lt 360777252868 2026-02-22T20:39:39.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:39.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089346 2026-02-22T20:39:39.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:39.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:39:39.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089346 2026-02-22T20:39:39.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:39.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089346 2026-02-22T20:39:39.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089346' 2026-02-22T20:39:39.828 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 382252089346 2026-02-22T20:39:39.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:39.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 382252089346 2026-02-22T20:39:39.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:40.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:39:40.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:41.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 382252089346 2026-02-22T20:39:41.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:42.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:39:42.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:42.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089346 -lt 382252089346 2026-02-22T20:39:42.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:42.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743432 2026-02-22T20:39:42.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:42.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:39:42.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743432 2026-02-22T20:39:42.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:42.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743432 2026-02-22T20:39:42.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743432' 2026-02-22T20:39:42.317 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 296352743432 2026-02-22T20:39:42.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:39:42.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743433 -lt 296352743432 2026-02-22T20:39:42.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:39:42.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:42.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:42.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:39:42.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:39:42.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:39:42.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:39:42.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:39:42.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:39:42.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:39:42.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:39:42.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:39:42.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:39:42.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:42.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:43.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:39:43.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:39:43.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:39:43.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3752: corrupt_scrub_erasure: for i in $(seq 1 $total_objs) 2026-02-22T20:39:43.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3753: corrupt_scrub_erasure: objname=EOBJ7 2026-02-22T20:39:43.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3754: corrupt_scrub_erasure: add_something td/osd-scrub-repair ecpool EOBJ7 2026-02-22T20:39:43.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:39:43.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:39:43.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=EOBJ7 2026-02-22T20:39:43.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:39:43.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:39:43.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:39:43.257 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:39:43.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:39:43.459 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:39:43.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:39:43.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:39:43.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:39:43.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: expr 7 % 2 2026-02-22T20:39:43.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3756: corrupt_scrub_erasure: local osd=1 2026-02-22T20:39:43.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3758: corrupt_scrub_erasure: case $i in 2026-02-22T20:39:43.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3803: corrupt_scrub_erasure: local payload=MAKETHISDIFFERENTFROMOTHEROBJECTS 2026-02-22T20:39:43.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3804: corrupt_scrub_erasure: echo MAKETHISDIFFERENTFROMOTHEROBJECTS 2026-02-22T20:39:43.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3805: corrupt_scrub_erasure: rados --pool ecpool put EOBJ7 td/osd-scrub-repair/DIFFERENT 2026-02-22T20:39:43.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3808: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-22T20:39:43.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:39:43.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:39:43.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:39:43.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:39:43.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-22T20:39:43.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:39:43.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:39:43.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:39:43.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:39:43.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:39:43.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:39:43.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:39:43.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:39:43.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:39:43.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:39:43.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:39:43.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ1 get-attr hinfo_key 2026-02-22T20:39:43.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:39:43.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:39:43.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:39:43.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:39:43.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:39:43.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ1 get-attr hinfo_key 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:39:44.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:39:44.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:39:44.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:39:44.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:39:44.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:39:44.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:39:44.001 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:39:44.001 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:39:44.001 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:39:44.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:44.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:44.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:39:44.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:39:44.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:39:44.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:39:44.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:39:44.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:39:44.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:39:44.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:39:44.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:39:44.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:39:44.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:39:44.003 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:39:44.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:39:44.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:39:44.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:39:44.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:39:44.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:39:44.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:39:44.021 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:44.010+0000 7f9d6dcbda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:44.023 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:44.010+0000 7f9d6dcbda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:44.024 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:44.010+0000 7f9d6dcbda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:44.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:39:44.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:39:44.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:39:44.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:39:44.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:39:44.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:44.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:39:44.168 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:39:44.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:44.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:39:44.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:39:44.470 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:44.458+0000 7f9d6dcbda40 -1 Falling back to public interface 2026-02-22T20:39:45.114 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:45.102+0000 7f9d6dcbda40 -1 osd.0 92 log_to_monitors true 2026-02-22T20:39:45.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:39:45.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:45.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:39:45.330 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:45.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:45.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:39:45.487 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 96 up_thru 96 down_at 93 last_clean_interval [84,92) [v2:127.0.0.1:6802/3702069099,v1:127.0.0.1:6803/3702069099] [v2:127.0.0.1:6804/3702069099,v1:127.0.0.1:6805/3702069099] exists,up 03e639bb-21c5-4e97-9f91-796824bfaceb 2026-02-22T20:39:45.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:39:45.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:39:45.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:39:45.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:39:45.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:39:45.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:39:45.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:39:45.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:39:45.488 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:39:45.488 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:39:45.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:39:45.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:39:45.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:39:45.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:39:45.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:39:45.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:39:45.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:39:45.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:39:45.603 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:39:45.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:39:45.768 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:45.768 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:39:45.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:39:45.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:45.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:39:45.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=412316860418 2026-02-22T20:39:45.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 412316860418 2026-02-22T20:39:45.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-412316860418' 2026-02-22T20:39:45.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:45.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:39:45.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089348 2026-02-22T20:39:45.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089348 2026-02-22T20:39:45.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-412316860418 1-382252089348' 2026-02-22T20:39:45.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:45.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:39:45.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743434 2026-02-22T20:39:45.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743434 2026-02-22T20:39:45.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-412316860418 1-382252089348 2-296352743434' 2026-02-22T20:39:45.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:45.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-412316860418 2026-02-22T20:39:45.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:45.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:39:45.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-412316860418 2026-02-22T20:39:45.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:45.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=412316860418 2026-02-22T20:39:45.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 412316860418' 2026-02-22T20:39:45.987 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 412316860418 2026-02-22T20:39:45.987 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:46.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 412316860418 2026-02-22T20:39:46.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:47.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:39:47.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:47.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 412316860418 2026-02-22T20:39:47.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:48.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:39:48.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:48.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 412316860418 -lt 412316860418 2026-02-22T20:39:48.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:48.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089348 2026-02-22T20:39:48.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:48.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:39:48.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089348 2026-02-22T20:39:48.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:48.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089348 2026-02-22T20:39:48.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089348' 2026-02-22T20:39:48.485 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 382252089348 2026-02-22T20:39:48.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:48.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089348 -lt 382252089348 2026-02-22T20:39:48.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:48.642 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743434 2026-02-22T20:39:48.642 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:48.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:39:48.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743434 2026-02-22T20:39:48.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:48.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743434 2026-02-22T20:39:48.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743434' 2026-02-22T20:39:48.646 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 296352743434 2026-02-22T20:39:48.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:39:48.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743435 -lt 296352743434 2026-02-22T20:39:48.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:39:48.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:48.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:49.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:39:49.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:39:49.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:39:49.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:39:49.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:39:49.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:39:49.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:39:49.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:39:49.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:39:49.167 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:39:49.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:49.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:49.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:39:49.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:39:49.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:39:49.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3809: corrupt_scrub_erasure: objectstore_tool td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-22T20:39:49.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:39:49.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:39:49.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:39:49.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:39:49.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-22T20:39:49.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:39:49.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:39:49.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:39:49.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:39:49.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:39:49.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:39:49.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:39:49.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:39:49.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:39:49.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:39:49.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:39:49.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-22T20:39:49.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:39:49.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:39:49.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:39:49.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:39:49.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:39:49.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 EOBJ7 set-attr hinfo_key td/osd-scrub-repair/hinfo 2026-02-22T20:39:49.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:39:49.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:39:49.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:39:49.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:39:49.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:39:49.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:39:49.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:39:49.952 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:39:49.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:39:49.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:39:49.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:39:49.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:39:49.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:39:49.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:39:49.972 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:49.958+0000 7f19106b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:49.973 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:49.962+0000 7f19106b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:49.974 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:49.962+0000 7f19106b6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:39:50.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:39:50.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:39:50.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:39:50.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:39:50.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:39:50.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:50.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:39:50.116 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:39:50.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:50.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:39:50.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:39:50.394 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:50.382+0000 7f19106b6a40 -1 Falling back to public interface 2026-02-22T20:39:51.021 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:39:51.010+0000 7f19106b6a40 -1 osd.0 98 log_to_monitors true 2026-02-22T20:39:51.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:39:51.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:39:51.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:39:51.279 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:51.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:39:51.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:39:51.436 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 102 up_thru 102 down_at 99 last_clean_interval [96,98) [v2:127.0.0.1:6802/3035557899,v1:127.0.0.1:6803/3035557899] [v2:127.0.0.1:6804/3035557899,v1:127.0.0.1:6805/3035557899] exists,up 03e639bb-21c5-4e97-9f91-796824bfaceb 2026-02-22T20:39:51.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:39:51.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:39:51.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:39:51.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:39:51.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:39:51.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:39:51.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:39:51.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:39:51.438 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:39:51.438 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:39:51.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:39:51.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:39:51.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:39:51.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:39:51.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:39:51.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:39:51.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:39:51.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:39:51.552 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:39:51.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:39:51.713 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:51.713 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:39:51.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:39:51.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:51.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:39:51.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=438086664194 2026-02-22T20:39:51.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 438086664194 2026-02-22T20:39:51.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664194' 2026-02-22T20:39:51.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:51.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:39:51.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089350 2026-02-22T20:39:51.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089350 2026-02-22T20:39:51.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664194 1-382252089350' 2026-02-22T20:39:51.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:51.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:39:51.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743437 2026-02-22T20:39:51.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743437 2026-02-22T20:39:51.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664194 1-382252089350 2-296352743437' 2026-02-22T20:39:51.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:51.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-438086664194 2026-02-22T20:39:51.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:51.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:39:51.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-438086664194 2026-02-22T20:39:51.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:51.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=438086664194 2026-02-22T20:39:51.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 438086664194' 2026-02-22T20:39:51.939 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 438086664194 2026-02-22T20:39:51.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:52.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 438086664194 2026-02-22T20:39:52.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:53.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:39:53.107 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:53.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 438086664194 2026-02-22T20:39:53.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:54.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:39:54.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:54.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 438086664194 -lt 438086664194 2026-02-22T20:39:54.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:54.428 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089350 2026-02-22T20:39:54.428 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:54.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:39:54.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089350 2026-02-22T20:39:54.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:54.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089350 2026-02-22T20:39:54.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089350' 2026-02-22T20:39:54.432 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 382252089350 2026-02-22T20:39:54.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:54.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089350 -lt 382252089350 2026-02-22T20:39:54.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:54.590 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743437 2026-02-22T20:39:54.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:54.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:39:54.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743437 2026-02-22T20:39:54.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:54.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743437 2026-02-22T20:39:54.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743437' 2026-02-22T20:39:54.594 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 296352743437 2026-02-22T20:39:54.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:39:54.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743437 -lt 296352743437 2026-02-22T20:39:54.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:39:54.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:54.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:54.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:39:54.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:39:54.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:39:54.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:39:54.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:39:54.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:39:54.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:39:54.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:39:55.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:39:55.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:39:55.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:39:55.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:39:55.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:39:55.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:39:55.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:39:55.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3810: corrupt_scrub_erasure: rm -f td/osd-scrub-repair/hinfo 2026-02-22T20:39:55.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3816: corrupt_scrub_erasure: get_pg ecpool EOBJ0 2026-02-22T20:39:55.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:39:55.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=EOBJ0 2026-02-22T20:39:55.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool EOBJ0 2026-02-22T20:39:55.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:39:55.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3816: corrupt_scrub_erasure: local pg=3.0 2026-02-22T20:39:55.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3818: corrupt_scrub_erasure: pg_scrub 3.0 2026-02-22T20:39:55.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-22T20:39:55.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-22T20:39:55.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:39:55.459 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:39:55.459 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:39:55.459 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:39:55.459 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:39:55.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:39:55.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:39:55.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:39:55.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:39:55.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:39:55.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:39:55.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:39:55.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:39:55.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:39:55.913 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:39:55.913 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:39:55.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:39:55.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:55.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:39:55.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=438086664195 2026-02-22T20:39:55.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 438086664195 2026-02-22T20:39:55.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664195' 2026-02-22T20:39:55.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:55.988 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:39:56.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089352 2026-02-22T20:39:56.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089352 2026-02-22T20:39:56.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664195 1-382252089352' 2026-02-22T20:39:56.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:39:56.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:39:56.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743438 2026-02-22T20:39:56.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743438 2026-02-22T20:39:56.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664195 1-382252089352 2-296352743438' 2026-02-22T20:39:56.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:56.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-438086664195 2026-02-22T20:39:56.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:56.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:39:56.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-438086664195 2026-02-22T20:39:56.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:56.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=438086664195 2026-02-22T20:39:56.136 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 438086664195 2026-02-22T20:39:56.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 438086664195' 2026-02-22T20:39:56.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:56.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 438086664194 -lt 438086664195 2026-02-22T20:39:56.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:57.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:39:57.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:57.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 438086664194 -lt 438086664195 2026-02-22T20:39:57.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:39:58.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:39:58.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:39:58.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 438086664196 -lt 438086664195 2026-02-22T20:39:58.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:58.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089352 2026-02-22T20:39:58.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:58.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:39:58.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089352 2026-02-22T20:39:58.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:58.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089352 2026-02-22T20:39:58.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089352' 2026-02-22T20:39:58.640 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 382252089352 2026-02-22T20:39:58.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:39:58.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089352 -lt 382252089352 2026-02-22T20:39:58.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:39:58.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743438 2026-02-22T20:39:58.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:39:58.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:39:58.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743438 2026-02-22T20:39:58.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:39:58.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743438 2026-02-22T20:39:58.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743438' 2026-02-22T20:39:58.809 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 296352743438 2026-02-22T20:39:58.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:39:58.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743439 -lt 296352743438 2026-02-22T20:39:58.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:39:58.968 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:39:58.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:39:58.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:39:58.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:39:58.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:39:58.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:39:58.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:39:59.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T20:39:59.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:39:59.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:39:59.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:39:59.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-22T20:39:59.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:39:59.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:39:59.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:39:59.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:39:59.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-22T20:38:43.600931+0000 2026-02-22T20:39:59.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-22T20:39:59.342 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0s0 on osd.1 to scrub 2026-02-22T20:39:59.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-22T20:38:43.600931+0000 2026-02-22T20:39:59.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:39:59.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:38:43.600931+0000 2026-02-22T20:39:59.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:39:59.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:39:59.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:39:59.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:39:59.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:39:59.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:39:59.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:39:59.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:39:59.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:43.600931+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:39:59.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:00.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:00.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:00.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:40:00.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:00.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:40:00.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:00.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:40:00.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:43.600931+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:00.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:01.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:01.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:01.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:40:01.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:01.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:40:01.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:01.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:40:01.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:43.600931+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:01.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:02.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:02.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:02.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:40:02.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:02.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:40:02.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:40:02.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:02.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:43.600931+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:02.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:03.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:03.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:03.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:40:03.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:03.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:40:03.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:40:03.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:04.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:43.600931+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:04.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:05.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:05.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:05.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:40:05.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:05.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:40:05.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:05.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:40:05.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:43.600931+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:05.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:06.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:06.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:06.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:40:06.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:06.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:40:06.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:06.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:40:06.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:39:59.861928+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:06.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:40:06.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3820: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2026-02-22T20:40:06.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3822: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2026-02-22T20:40:06.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3822: corrupt_scrub_erasure: test 1 = 1 2026-02-22T20:40:06.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3824: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-22T20:40:06.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3824: corrupt_scrub_erasure: test 3.0 = 3.0 2026-02-22T20:40:06.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3826: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2026-02-22T20:40:06.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3828: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2026-02-22T20:40:06.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3828: corrupt_scrub_erasure: epoch=102 2026-02-22T20:40:06.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-22T20:40:06.541 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:40:06.541 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:40:06.541 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:40:06.541 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:40:06.541 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:40:06.541 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:40:06.541 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:40:06.541 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:40:06.541 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:40:06.542 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:40:06.542 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:40:06.542 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:40:06.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: jq .inconsistents 2026-02-22T20:40:06.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3830: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:40:06.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-22T20:40:06.559 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:40:06.559 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:40:06.560 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:40:06.560 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:40:06.560 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:40:06.560 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:40:06.560 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:40:06.560 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:40:06.560 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:40:06.560 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:40:06.560 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:40:06.560 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-22T20:40:06.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: jq .inconsistents 2026-02-22T20:40:06.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4458: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:40:06.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4459: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:40:06.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:40:06.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4460: corrupt_scrub_erasure: test no = yes 2026-02-22T20:40:06.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4465: corrupt_scrub_erasure: test '' = yes 2026-02-22T20:40:06.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4470: corrupt_scrub_erasure: pg_deep_scrub 3.0 2026-02-22T20:40:06.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-22T20:40:06.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-22T20:40:06.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:40:06.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:40:06.585 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:40:06.585 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:40:06.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:40:06.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:40:06.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:40:06.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:40:06.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:40:06.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:40:06.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:40:06.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:40:06.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:40:07.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:40:07.044 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:40:07.044 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:40:07.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:40:07.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:07.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:40:07.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=438086664199 2026-02-22T20:40:07.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 438086664199 2026-02-22T20:40:07.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664199' 2026-02-22T20:40:07.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:07.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:40:07.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=382252089355 2026-02-22T20:40:07.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 382252089355 2026-02-22T20:40:07.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664199 1-382252089355' 2026-02-22T20:40:07.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:07.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:40:07.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=296352743442 2026-02-22T20:40:07.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 296352743442 2026-02-22T20:40:07.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-438086664199 1-382252089355 2-296352743442' 2026-02-22T20:40:07.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:40:07.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-438086664199 2026-02-22T20:40:07.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:40:07.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:40:07.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-438086664199 2026-02-22T20:40:07.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:40:07.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=438086664199 2026-02-22T20:40:07.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 438086664199' 2026-02-22T20:40:07.267 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 438086664199 2026-02-22T20:40:07.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:40:07.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 438086664197 -lt 438086664199 2026-02-22T20:40:07.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:40:08.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:40:08.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:40:08.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 438086664199 -lt 438086664199 2026-02-22T20:40:08.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:40:08.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-382252089355 2026-02-22T20:40:08.582 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:40:08.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:40:08.584 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-382252089355 2026-02-22T20:40:08.584 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:40:08.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=382252089355 2026-02-22T20:40:08.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 382252089355' 2026-02-22T20:40:08.586 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 382252089355 2026-02-22T20:40:08.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:40:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 382252089355 -lt 382252089355 2026-02-22T20:40:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:40:08.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-296352743442 2026-02-22T20:40:08.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:40:08.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:40:08.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-296352743442 2026-02-22T20:40:08.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:40:08.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=296352743442 2026-02-22T20:40:08.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 296352743442' 2026-02-22T20:40:08.756 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 296352743442 2026-02-22T20:40:08.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:40:08.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 296352743442 -lt 296352743442 2026-02-22T20:40:08.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:40:08.927 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:40:08.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:40:08.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:40:08.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:40:08.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:40:08.927 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:40:08.927 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:40:09.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-22T20:40:09.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:40:09.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:40:09.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:40:09.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:09.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:09.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:09.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:09.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:09.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:09.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-22T20:40:09.308 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0s0 on osd.1 to deep-scrub 2026-02-22T20:40:09.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-22T20:38:43.600931+0000 last_deep_scrub_stamp 2026-02-22T20:40:09.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:40:09.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:09.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-22T20:40:09.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:40:09.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:09.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:09.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:09.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:09.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:09.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:09.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:43.600931+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:09.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:10.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:10.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:10.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:10.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:10.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:10.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:10.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:10.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:43.600931+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:10.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:11.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:11.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:11.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:11.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:11.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:11.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:11.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:11.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:43.600931+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:11.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:12.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:12.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:12.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:12.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:12.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:12.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:12.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:12.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:43.600931+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:12.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:13.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:13.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:13.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:13.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:13.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:13.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:13.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:14.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:43.600931+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:14.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:15.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:15.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:15.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:15.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:15.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:15.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:15.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:15.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:38:43.600931+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:15.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:16.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:16.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:16.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:16.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:16.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:16.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:16.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:16.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:09.871243+0000 '>' 2026-02-22T20:38:43.600931+0000 2026-02-22T20:40:16.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:40:16.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4472: corrupt_scrub_erasure: rados list-inconsistent-pg ecpool 2026-02-22T20:40:16.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4474: corrupt_scrub_erasure: jq '. | length' td/osd-scrub-repair/json 2026-02-22T20:40:16.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4474: corrupt_scrub_erasure: test 1 = 1 2026-02-22T20:40:16.472 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4476: corrupt_scrub_erasure: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-22T20:40:16.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4476: corrupt_scrub_erasure: test 3.0 = 3.0 2026-02-22T20:40:16.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4478: corrupt_scrub_erasure: rados list-inconsistent-obj 3.0 2026-02-22T20:40:16.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4479: corrupt_scrub_erasure: cp td/osd-scrub-repair/json /tmp/rrr6.json 2026-02-22T20:40:16.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4481: corrupt_scrub_erasure: jq .epoch td/osd-scrub-repair/json 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4481: corrupt_scrub_erasure: epoch=102 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4483: corrupt_scrub_erasure: '[' true = true ']' 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4485: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:40:16.515 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:40:16.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4485: corrupt_scrub_erasure: jq .inconsistents 2026-02-22T20:40:16.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:4485: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:40:16.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: jq 'def walk(f): 2026-02-22T20:40:16.533 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:40:16.533 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:40:16.534 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:40:16.534 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:40:16.534 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:40:16.534 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:40:16.534 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:40:16.534 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:40:16.534 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:40:16.534 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:40:16.534 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:40:16.534 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-22T20:40:16.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: jq .inconsistents 2026-02-22T20:40:16.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5890: corrupt_scrub_erasure: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:40:16.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5891: corrupt_scrub_erasure: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:40:16.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:40:16.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5892: corrupt_scrub_erasure: test no = yes 2026-02-22T20:40:16.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5903: corrupt_scrub_erasure: test '' = yes 2026-02-22T20:40:16.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5908: corrupt_scrub_erasure: ceph osd pool rm ecpool ecpool --yes-i-really-really-mean-it 2026-02-22T20:40:16.764 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' removed 2026-02-22T20:40:16.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:40:16.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:40:16.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:40:16.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:40:16.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:40:16.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:40:16.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:40:16.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:40:16.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:40:16.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:40:16.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:40:16.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:40:16.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:40:16.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:40:16.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:40:16.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:40:16.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:40:16.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:40:16.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:40:16.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:40:16.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:40:16.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:40:16.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:40:16.923 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:40:16.924 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:16.924 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:16.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:40:16.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:40:16.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:40:16.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:40:16.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:40:16.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:40:16.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:40:16.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:40:16.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:40:16.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:40:16.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:40:16.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:40:16.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:40:16.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:40:16.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:40:16.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:40:16.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:40:16.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:40:16.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:40:16.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:40:16.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:40:16.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:40:16.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:40:16.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:40:16.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:40:16.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:40:16.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:40:16.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:40:16.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:40:16.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:40:16.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:16.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:16.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:40:16.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:40:16.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:40:16.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:40:16.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:40:16.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:16.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:16.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:40:16.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:40:16.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:40:16.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:40:16.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:40:16.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_scrub_replicated td/osd-scrub-repair 2026-02-22T20:40:16.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1050: TEST_corrupt_scrub_replicated: local dir=td/osd-scrub-repair 2026-02-22T20:40:16.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1051: TEST_corrupt_scrub_replicated: local poolname=csr_pool 2026-02-22T20:40:16.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1052: TEST_corrupt_scrub_replicated: local total_objs=19 2026-02-22T20:40:16.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1054: TEST_corrupt_scrub_replicated: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-22T20:40:16.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:40:16.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:40:16.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:40:16.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:40:16.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:40:16.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-22T20:40:16.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:40:16.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:40:16.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:40:16.968 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:40:16.968 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:16.968 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:16.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:40:16.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-22T20:40:16.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:40:16.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:40:16.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:40:16.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:40:16.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:40:16.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:40:16.998 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:40:16.998 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:40:16.998 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:40:16.999 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:40:16.999 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:16.999 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:16.999 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:40:17.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:40:17.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:40:17.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:40:17.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:40:17.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:40:17.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:40:17.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:40:17.054 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:40:17.054 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:40:17.054 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:40:17.054 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:40:17.055 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:17.055 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:17.055 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:40:17.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:40:17.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:40:17.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1055: TEST_corrupt_scrub_replicated: run_mgr td/osd-scrub-repair x 2026-02-22T20:40:17.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:40:17.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:40:17.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:40:17.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:40:17.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:40:17.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:40:17.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:40:17.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:40:17.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:40:17.208 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:40:17.208 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:17.208 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:17.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:40:17.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:40:17.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:40:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1056: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 0 2026-02-22T20:40:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:40:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:40:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:40:17.231 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:40:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:40:17.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:40:17.233 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:40:17.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:40:17.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 fd75825d-4113-41d3-8c2e-0ab21035c87d' 2026-02-22T20:40:17.234 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:40:17.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:40:17.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCxaZtpw6lTDhAAIkGhI93JKQmDE5fihAJQJw== 2026-02-22T20:40:17.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCxaZtpw6lTDhAAIkGhI93JKQmDE5fihAJQJw=="}' 2026-02-22T20:40:17.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fd75825d-4113-41d3-8c2e-0ab21035c87d -i td/osd-scrub-repair/0/new.json 2026-02-22T20:40:17.339 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:40:17.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:40:17.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxaZtpw6lTDhAAIkGhI93JKQmDE5fihAJQJw== --osd-uuid fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:40:17.366 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:17.354+0000 7f331378fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:17.367 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:17.354+0000 7f331378fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:17.368 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:17.358+0000 7f331378fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:17.368 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:17.358+0000 7f331378fa40 -1 bdev(0x5611c0944c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:40:17.368 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:17.358+0000 7f331378fa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:40:18.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:40:18.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:40:18.097 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:40:18.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:40:18.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:40:18.300 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:40:18.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:40:18.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:40:18.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:40:18.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:40:18.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:40:18.324 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:18.314+0000 7ff89a155a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:18.326 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:18.314+0000 7ff89a155a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:18.327 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:18.314+0000 7ff89a155a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:18.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:40:18.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:40:18.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:40:18.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:40:18.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:40:18.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:40:18.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:40:18.464 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:40:18.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:40:18.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:40:18.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:40:18.734 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:18.722+0000 7ff89a155a40 -1 Falling back to public interface 2026-02-22T20:40:19.285 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:19.274+0000 7ff89a155a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:40:19.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:40:19.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:40:19.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:40:19.621 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:40:19.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:40:19.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:40:19.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:40:20.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:40:20.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:40:20.778 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:40:20.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:40:20.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:40:20.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:40:20.936 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2192993112,v1:127.0.0.1:6803/2192993112] [v2:127.0.0.1:6804/2192993112,v1:127.0.0.1:6805/2192993112] exists,up fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1057: TEST_corrupt_scrub_replicated: run_osd td/osd-scrub-repair 1 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:40:20.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:40:20.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:40:20.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:40:20.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:40:20.938 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:40:20.938 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:20.938 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:20.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:40:20.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:40:20.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:40:20.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:40:20.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:40:20.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:40:20.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:40:20.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:40:20.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:40:20.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:40:20.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:40:20.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:40:20.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:40:20.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:40:20.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:40:20.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:40:20.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 9384a6a4-a814-4e77-8a87-47e2d6def999' 2026-02-22T20:40:20.941 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:40:20.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:40:20.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC0aZtpncBTOBAAGs6jcx+VI6pDPRuET3hTLw== 2026-02-22T20:40:20.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC0aZtpncBTOBAAGs6jcx+VI6pDPRuET3hTLw=="}' 2026-02-22T20:40:20.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9384a6a4-a814-4e77-8a87-47e2d6def999 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:40:21.116 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:40:21.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:40:21.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0aZtpncBTOBAAGs6jcx+VI6pDPRuET3hTLw== --osd-uuid 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:40:21.142 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:21.130+0000 7f4b97c4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:21.144 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:21.134+0000 7f4b97c4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:21.145 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:21.134+0000 7f4b97c4ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:21.145 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:21.134+0000 7f4b97c4ea40 -1 bdev(0x55b58ce7cc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:40:21.146 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:21.134+0000 7f4b97c4ea40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:40:21.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:40:21.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:40:21.833 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:40:21.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:40:21.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:40:22.040 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:40:22.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:40:22.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:40:22.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:40:22.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:40:22.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:40:22.056 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:22.042+0000 7fbbc91f5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:22.057 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:22.046+0000 7fbbc91f5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:22.058 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:22.046+0000 7fbbc91f5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:22.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:40:22.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:40:22.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:40:22.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:40:22.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:40:22.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:40:22.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:40:22.200 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:40:22.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:40:22.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:40:22.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:40:22.450 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:22.438+0000 7fbbc91f5a40 -1 Falling back to public interface 2026-02-22T20:40:23.050 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:23.038+0000 7fbbc91f5a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:40:23.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:40:23.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:40:23.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:40:23.364 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:40:23.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:40:23.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:40:23.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:40:24.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:40:24.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:40:24.530 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:40:24.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:40:24.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:40:24.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:40:24.687 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/663723716,v1:127.0.0.1:6811/663723716] [v2:127.0.0.1:6812/663723716,v1:127.0.0.1:6813/663723716] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:40:24.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:40:24.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:40:24.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:40:24.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1058: TEST_corrupt_scrub_replicated: create_rbd_pool 2026-02-22T20:40:24.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:40:24.842 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:40:24.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:40:24.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:40:25.061 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:40:25.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:40:26.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:40:26.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1059: TEST_corrupt_scrub_replicated: wait_for_clean 2026-02-22T20:40:26.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:40:26.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:40:26.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:40:26.353 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:40:26.353 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:40:26.353 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:40:26.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:40:26.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:40:26.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:40:26.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:40:26.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:40:26.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:40:26.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:40:26.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:40:26.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:40:26.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:40:26.627 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:40:26.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:40:26.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:26.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:40:26.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:40:26.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:40:26.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:40:26.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:26.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:40:26.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672962 2026-02-22T20:40:26.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672962 2026-02-22T20:40:26.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672962' 2026-02-22T20:40:26.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:40:26.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:40:26.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:40:26.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:40:26.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:40:26.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:40:26.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:40:26.777 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:40:26.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:40:26.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:40:26.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:40:26.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:40:27.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:40:27.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:40:28.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T20:40:28.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:40:28.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672962 2026-02-22T20:40:28.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:40:28.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:40:28.099 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672962 2026-02-22T20:40:28.100 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:40:28.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672962 2026-02-22T20:40:28.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2026-02-22T20:40:28.102 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672962 2026-02-22T20:40:28.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:40:28.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672962 -lt 42949672962 2026-02-22T20:40:28.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:40:28.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:40:28.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:40:28.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:40:28.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:40:28.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:40:28.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:40:28.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:40:28.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:40:28.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:40:28.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:40:28.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:40:28.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:40:28.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:40:28.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:40:28.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:40:28.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:40:28.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:40:28.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1061: TEST_corrupt_scrub_replicated: create_pool foo 1 2026-02-22T20:40:28.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-22T20:40:29.015 INFO:tasks.workunit.client.0.trial150.stderr:pool 'foo' created 2026-02-22T20:40:29.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:40:30.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1062: TEST_corrupt_scrub_replicated: create_pool csr_pool 1 1 2026-02-22T20:40:30.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2026-02-22T20:40:30.236 INFO:tasks.workunit.client.0.trial150.stderr:pool 'csr_pool' created 2026-02-22T20:40:30.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:40:31.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1063: TEST_corrupt_scrub_replicated: wait_for_clean 2026-02-22T20:40:31.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:40:31.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:40:31.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:40:31.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:40:31.251 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:40:31.251 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:40:31.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:40:31.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:40:31.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:40:31.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:40:31.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:40:31.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:40:31.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:40:31.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:40:31.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:40:31.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:40:31.525 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:40:31.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:40:31.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:31.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:40:31.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-22T20:40:31.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-22T20:40:31.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-22T20:40:31.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:31.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:40:31.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-22T20:40:31.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-22T20:40:31.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672964' 2026-02-22T20:40:31.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:40:31.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-22T20:40:31.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:40:31.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:40:31.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-22T20:40:31.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:40:31.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-22T20:40:31.676 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836485 2026-02-22T20:40:31.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-22T20:40:31.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:40:31.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-22T20:40:31.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:40:31.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-22T20:40:31.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:40:31.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:40:31.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672964 2026-02-22T20:40:31.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:40:31.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672964 2026-02-22T20:40:31.852 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672964 2026-02-22T20:40:31.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-22T20:40:31.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:40:32.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672964 2026-02-22T20:40:32.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:40:32.023 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:40:32.023 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:40:32.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:40:32.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:40:32.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:40:32.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:40:32.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:40:32.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:40:32.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:40:32.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:40:32.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:40:32.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:40:32.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:40:32.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:40:32.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:40:32.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:40:32.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:40:32.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: seq 1 19 2026-02-22T20:40:32.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:32.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ1 2026-02-22T20:40:32.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ1 2026-02-22T20:40:32.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:32.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:32.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2026-02-22T20:40:32.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:32.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:32.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:32.778 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:32.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:32.983 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:32.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:32.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:32.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:33.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2026-02-22T20:40:33.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2026-02-22T20:40:33.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:33.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ2 2026-02-22T20:40:33.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ2 2026-02-22T20:40:33.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:33.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:33.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2026-02-22T20:40:33.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:33.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:33.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:33.272 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:33.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:33.476 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:33.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:33.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:33.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:33.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2026-02-22T20:40:33.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2026-02-22T20:40:33.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:33.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ3 2026-02-22T20:40:33.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ3 2026-02-22T20:40:33.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:33.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:33.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2026-02-22T20:40:33.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:33.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:33.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:33.760 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:33.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:33.963 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:33.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:33.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:33.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:33.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2026-02-22T20:40:34.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2026-02-22T20:40:34.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:34.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ4 2026-02-22T20:40:34.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ4 2026-02-22T20:40:34.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:34.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:34.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2026-02-22T20:40:34.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:34.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:34.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:34.255 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:34.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:34.460 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:34.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:34.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:34.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:34.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2026-02-22T20:40:34.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2026-02-22T20:40:34.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:34.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ5 2026-02-22T20:40:34.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ5 2026-02-22T20:40:34.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:34.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:34.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2026-02-22T20:40:34.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:34.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:34.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:34.749 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:34.954 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:34.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:34.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:34.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:34.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2026-02-22T20:40:35.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2026-02-22T20:40:35.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:35.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ6 2026-02-22T20:40:35.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ6 2026-02-22T20:40:35.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:35.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:35.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2026-02-22T20:40:35.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:35.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:35.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:35.251 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:35.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:35.455 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:35.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:35.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:35.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:35.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2026-02-22T20:40:35.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2026-02-22T20:40:35.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:35.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ7 2026-02-22T20:40:35.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ7 2026-02-22T20:40:35.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:35.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:35.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2026-02-22T20:40:35.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:35.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:35.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:35.744 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:35.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:35.949 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:35.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:35.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:35.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:35.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2026-02-22T20:40:36.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2026-02-22T20:40:36.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:36.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ8 2026-02-22T20:40:36.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ8 2026-02-22T20:40:36.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:36.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:36.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2026-02-22T20:40:36.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:36.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:36.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:36.247 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:36.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:36.451 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:36.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:36.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:36.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:36.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2026-02-22T20:40:36.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2026-02-22T20:40:36.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:36.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ9 2026-02-22T20:40:36.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ9 2026-02-22T20:40:36.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:36.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:36.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2026-02-22T20:40:36.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:36.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:36.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:36.749 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:36.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:36.953 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:36.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:36.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:36.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:36.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2026-02-22T20:40:37.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2026-02-22T20:40:37.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:37.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ10 2026-02-22T20:40:37.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ10 2026-02-22T20:40:37.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:37.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:37.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2026-02-22T20:40:37.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:37.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:37.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:37.234 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:37.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:37.437 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:37.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:37.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:37.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:37.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2026-02-22T20:40:37.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2026-02-22T20:40:37.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:37.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ11 2026-02-22T20:40:37.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ11 2026-02-22T20:40:37.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:37.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:37.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2026-02-22T20:40:37.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:37.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:37.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:37.725 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:37.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:37.929 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:37.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:37.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:37.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:37.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2026-02-22T20:40:37.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2026-02-22T20:40:38.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:38.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ12 2026-02-22T20:40:38.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ12 2026-02-22T20:40:38.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:38.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:38.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2026-02-22T20:40:38.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:38.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:38.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:38.229 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:38.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:38.432 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:38.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:38.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:38.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:38.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2026-02-22T20:40:38.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2026-02-22T20:40:38.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:38.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ13 2026-02-22T20:40:38.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ13 2026-02-22T20:40:38.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:38.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:38.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2026-02-22T20:40:38.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:38.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:38.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:38.721 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:38.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:38.926 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:38.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:38.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:38.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:38.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2026-02-22T20:40:38.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2026-02-22T20:40:39.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:39.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ14 2026-02-22T20:40:39.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ14 2026-02-22T20:40:39.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:39.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:39.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2026-02-22T20:40:39.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:39.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:39.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:39.219 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:39.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:39.423 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:39.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:39.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:39.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:39.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2026-02-22T20:40:39.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2026-02-22T20:40:39.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:39.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ15 2026-02-22T20:40:39.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ15 2026-02-22T20:40:39.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:39.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:39.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2026-02-22T20:40:39.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:39.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:39.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:39.696 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:39.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:39.900 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:39.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:39.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:39.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:39.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2026-02-22T20:40:39.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2026-02-22T20:40:39.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:39.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ16 2026-02-22T20:40:39.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ16 2026-02-22T20:40:39.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:39.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:39.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2026-02-22T20:40:39.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:39.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:39.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:40.193 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:40.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:40.398 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:40.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:40.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:40.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:40.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2026-02-22T20:40:40.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2026-02-22T20:40:40.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:40.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ17 2026-02-22T20:40:40.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ17 2026-02-22T20:40:40.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:40.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:40.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2026-02-22T20:40:40.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:40.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:40.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:40.685 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:40.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:40.888 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:40.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:40.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:40.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:40.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2026-02-22T20:40:40.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2026-02-22T20:40:40.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:40.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ18 2026-02-22T20:40:40.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ18 2026-02-22T20:40:40.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:40.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:40.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2026-02-22T20:40:40.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:40.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:40.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:41.186 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:41.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:41.390 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:41.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:41.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:41.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:41.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2026-02-22T20:40:41.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2026-02-22T20:40:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1065: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1066: TEST_corrupt_scrub_replicated: objname=ROBJ19 2026-02-22T20:40:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1067: TEST_corrupt_scrub_replicated: add_something td/osd-scrub-repair csr_pool ROBJ19 2026-02-22T20:40:41.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:40:41.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:40:41.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2026-02-22T20:40:41.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:40:41.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:40:41.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:40:41.684 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:40:41.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:40:41.889 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:40:41.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:40:41.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:40:41.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:40:41.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1069: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2026-02-22T20:40:41.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1070: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2026-02-22T20:40:41.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1074: TEST_corrupt_scrub_replicated: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2026-02-22T20:40:41.974 INFO:tasks.workunit.client.0.trial150.stderr:1025+0 records in 2026-02-22T20:40:41.974 INFO:tasks.workunit.client.0.trial150.stderr:1025+0 records out 2026-02-22T20:40:41.974 INFO:tasks.workunit.client.0.trial150.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00408573 s, 257 MB/s 2026-02-22T20:40:41.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1075: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2026-02-22T20:40:42.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1076: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ19 2026-02-22T20:40:42.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1078: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2026-02-22T20:40:42.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-22T20:40:42.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-22T20:40:42.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-22T20:40:42.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:40:42.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1078: TEST_corrupt_scrub_replicated: local pg=3.0 2026-02-22T20:40:42.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1079: TEST_corrupt_scrub_replicated: get_primary csr_pool ROBJ0 2026-02-22T20:40:42.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2026-02-22T20:40:42.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2026-02-22T20:40:42.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2026-02-22T20:40:42.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:40:42.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1079: TEST_corrupt_scrub_replicated: local primary=1 2026-02-22T20:40:42.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1082: TEST_corrupt_scrub_replicated: get_asok_path osd.0 2026-02-22T20:40:42.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:40:42.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:40:42.323 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:40:42.323 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:42.324 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:42.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:40:42.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1082: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2026-02-22T20:40:42.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1082: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.40073/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2026-02-22T20:40:42.385 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:40:42.385 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2026-02-22T20:40:42.385 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:40:42.391 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: get_asok_path osd.1 2026-02-22T20:40:42.391 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:40:42.391 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:40:42.391 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:40:42.391 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:42.392 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:42.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:40:42.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: CEPH_ARGS= 2026-02-22T20:40:42.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1084: TEST_corrupt_scrub_replicated: ceph daemon /tmp/ceph-asok.40073/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2026-02-22T20:40:42.448 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:40:42.448 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2026-02-22T20:40:42.448 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:40:42.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1086: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2026-02-22T20:40:42.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-22T20:40:42.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-22T20:40:42.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:40:42.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:40:42.455 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:40:42.455 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:40:42.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:40:42.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:40:42.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:40:42.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:40:42.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:40:42.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:40:42.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:40:42.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:40:42.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:40:42.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:40:42.903 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:40:42.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:40:42.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:42.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:40:42.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-22T20:40:42.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-22T20:40:42.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-22T20:40:42.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:42.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:40:43.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-22T20:40:43.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-22T20:40:43.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2026-02-22T20:40:43.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:40:43.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-22T20:40:43.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:40:43.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:40:43.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-22T20:40:43.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:40:43.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-22T20:40:43.055 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836488 2026-02-22T20:40:43.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-22T20:40:43.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:40:43.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-22T20:40:43.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:40:44.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:40:44.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:40:44.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-22T20:40:44.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:40:44.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-22T20:40:44.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:40:44.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:40:44.387 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-22T20:40:44.387 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:40:44.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-22T20:40:44.389 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672967 2026-02-22T20:40:44.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-22T20:40:44.389 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:40:44.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672967 -lt 42949672967 2026-02-22T20:40:44.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:40:44.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:40:44.549 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:40:44.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:40:44.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:40:44.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:40:44.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:40:44.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:40:44.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T20:40:44.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:40:44.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:40:44.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:40:44.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:44.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:44.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:44.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:44.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:44.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-22T20:40:30.227414+0000 2026-02-22T20:40:44.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-22T20:40:44.930 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-22T20:40:44.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-22T20:40:30.227414+0000 last_deep_scrub_stamp 2026-02-22T20:40:44.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:40:44.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:40:30.227414+0000 2026-02-22T20:40:44.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-22T20:40:44.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:40:44.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:44.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:44.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:44.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:44.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:44.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:45.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:30.227414+0000 '>' 2026-02-22T20:40:30.227414+0000 2026-02-22T20:40:45.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:46.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:46.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:46.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:46.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:46.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:46.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:46.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:46.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:30.227414+0000 '>' 2026-02-22T20:40:30.227414+0000 2026-02-22T20:40:46.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:47.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:47.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:47.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:47.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:47.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:47.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:47.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:47.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:30.227414+0000 '>' 2026-02-22T20:40:30.227414+0000 2026-02-22T20:40:47.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:48.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:48.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:48.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:48.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:48.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:48.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:48.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:48.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:30.227414+0000 '>' 2026-02-22T20:40:30.227414+0000 2026-02-22T20:40:48.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:40:49.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:40:49.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:40:49.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:40:49.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:40:49.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:40:49.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:40:49.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:40:49.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:45.395897+0000 '>' 2026-02-22T20:40:30.227414+0000 2026-02-22T20:40:49.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:40:49.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: seq 1 19 2026-02-22T20:40:49.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:49.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ1 2026-02-22T20:40:49.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 1 % 2 2026-02-22T20:40:49.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-22T20:40:49.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:40:49.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1097: TEST_corrupt_scrub_replicated: local payload=UVWXYZZZ 2026-02-22T20:40:49.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1098: TEST_corrupt_scrub_replicated: echo UVWXYZZZ 2026-02-22T20:40:49.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1099: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:40:49.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:40:49.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:40:49.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:40:49.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:40:49.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:40:49.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:40:49.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:40:49.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:40:49.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:40:49.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:40:49.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:40:49.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:40:49.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:40:49.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:40:49.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:40:50.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:40:50.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:40:50.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:40:50.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:40:50.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:40:50.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:40:50.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:40:50.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:40:50.261 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#-5:00000000:::deep_scrub_3.0:head#, (61) No data available 2026-02-22T20:40:50.261 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2026-02-22T20:40:50.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:40:50.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:40:50.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:40:50.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:40:50.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:40:50.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:40:50.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:40:50.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:40:50.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:40:50.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:40:50.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:40:50.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:40:50.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:40:50.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:40:50.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:40:50.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:40:50.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:40:50.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:40:50.369 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:40:50.369 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:50.369 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:50.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:40:50.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:40:50.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:40:50.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:40:50.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:40:50.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:40:50.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:40:50.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:40:50.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:40:50.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:40:50.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:40:50.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:40:50.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:40:50.371 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:40:50.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:40:50.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:40:50.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:40:50.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:40:50.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:40:50.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:40:50.383 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:50.370+0000 7f93037aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:50.384 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:50.370+0000 7f93037aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:50.385 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:50.374+0000 7f93037aba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:50.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:40:50.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:40:50.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:40:50.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:40:50.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:40:50.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:40:50.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:40:50.528 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:40:50.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:40:50.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:40:50.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:40:50.802 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:50.790+0000 7f93037aba40 -1 Falling back to public interface 2026-02-22T20:40:51.388 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:51.378+0000 7f93037aba40 -1 osd.1 62 log_to_monitors true 2026-02-22T20:40:51.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:40:51.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:40:51.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:40:51.687 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:40:51.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:40:51.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:40:51.844 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 66 up_thru 66 down_at 63 last_clean_interval [10,62) [v2:127.0.0.1:6810/26565092,v1:127.0.0.1:6811/26565092] [v2:127.0.0.1:6812/26565092,v1:127.0.0.1:6813/26565092] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:40:51.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:40:51.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:40:51.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:40:51.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:40:51.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:40:51.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:40:51.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:40:51.845 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:40:51.846 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:40:51.846 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:40:51.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:40:51.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:40:51.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:40:51.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:40:51.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:40:51.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:40:51.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:40:51.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:40:51.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:40:52.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:40:52.124 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:40:52.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:40:52.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:52.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:40:52.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-22T20:40:52.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-22T20:40:52.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T20:40:52.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:52.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:40:52.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841538 2026-02-22T20:40:52.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841538 2026-02-22T20:40:52.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-283467841538' 2026-02-22T20:40:52.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:40:52.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-22T20:40:52.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:40:52.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:40:52.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-22T20:40:52.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:40:52.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-22T20:40:52.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T20:40:52.279 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836491 2026-02-22T20:40:52.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:40:52.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T20:40:52.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:40:53.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:40:53.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:40:53.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T20:40:53.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:40:54.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:40:54.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:40:54.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-22T20:40:54.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:40:54.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-283467841538 2026-02-22T20:40:54.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:40:54.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:40:54.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-283467841538 2026-02-22T20:40:54.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:40:54.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841538 2026-02-22T20:40:54.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 283467841538' 2026-02-22T20:40:54.766 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 283467841538 2026-02-22T20:40:54.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:40:54.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841538 -lt 283467841538 2026-02-22T20:40:54.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:40:54.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:40:54.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:40:55.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:40:55.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:40:55.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:40:55.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:40:55.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:40:55.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:40:55.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:40:55.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:40:55.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:40:55.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:40:55.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:40:55.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:40:55.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:40:55.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:40:55.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:40:55.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:40:55.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ2 2026-02-22T20:40:55.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 2 % 2 2026-02-22T20:40:55.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-22T20:40:55.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:40:55.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1104: TEST_corrupt_scrub_replicated: local payload=UVWXYZ 2026-02-22T20:40:55.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1105: TEST_corrupt_scrub_replicated: echo UVWXYZ 2026-02-22T20:40:55.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1106: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:40:55.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:40:55.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:40:55.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:40:55.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:40:55.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:40:55.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:40:55.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:40:55.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:40:55.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:40:55.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:40:55.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:40:55.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:40:55.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:40:55.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:40:55.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:40:55.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:40:55.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:40:55.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:40:55.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:40:55.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:40:55.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:40:55.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:40:55.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:40:56.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:40:56.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:40:56.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:40:56.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:40:56.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:40:56.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:40:56.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:40:56.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:40:56.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:40:56.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:40:56.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:40:56.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:40:56.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:40:56.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:40:56.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:40:56.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:40:56.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:40:56.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:40:56.111 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:40:56.111 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:40:56.111 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:40:56.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:40:56.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:40:56.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:40:56.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:40:56.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:40:56.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:40:56.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:40:56.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:40:56.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:40:56.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:40:56.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:40:56.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:40:56.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:40:56.112 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:40:56.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:40:56.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:40:56.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:40:56.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:40:56.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:40:56.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:40:56.131 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:56.118+0000 7fdb4b55fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:56.133 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:56.122+0000 7fdb4b55fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:56.134 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:56.122+0000 7fdb4b55fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:40:56.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:40:56.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:40:56.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:40:56.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:40:56.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:40:56.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:40:56.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:40:56.275 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:40:56.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:40:56.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:40:56.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:40:56.550 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:56.538+0000 7fdb4b55fa40 -1 Falling back to public interface 2026-02-22T20:40:57.109 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:40:57.098+0000 7fdb4b55fa40 -1 osd.0 69 log_to_monitors true 2026-02-22T20:40:57.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:40:57.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:40:57.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:40:57.436 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:40:57.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:40:57.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:40:57.593 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 72 up_thru 72 down_at 69 last_clean_interval [5,69) [v2:127.0.0.1:6802/3547916903,v1:127.0.0.1:6803/3547916903] [v2:127.0.0.1:6804/3547916903,v1:127.0.0.1:6805/3547916903] exists,up fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:40:57.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:40:57.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:40:57.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:40:57.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:40:57.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:40:57.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:40:57.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:40:57.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:40:57.593 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:40:57.594 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:40:57.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:40:57.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:40:57.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:40:57.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:40:57.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:40:57.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:40:57.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:40:57.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:40:57.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:40:57.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:40:57.866 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:40:57.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:40:57.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:57.866 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:40:57.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=309237645314 2026-02-22T20:40:57.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 309237645314 2026-02-22T20:40:57.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-309237645314' 2026-02-22T20:40:57.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:40:57.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:40:58.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=283467841540 2026-02-22T20:40:58.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 283467841540 2026-02-22T20:40:58.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-309237645314 1-283467841540' 2026-02-22T20:40:58.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:40:58.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-309237645314 2026-02-22T20:40:58.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:40:58.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:40:58.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-309237645314 2026-02-22T20:40:58.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:40:58.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=309237645314 2026-02-22T20:40:58.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 309237645314' 2026-02-22T20:40:58.016 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 309237645314 2026-02-22T20:40:58.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:40:58.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 309237645314 2026-02-22T20:40:58.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:40:59.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:40:59.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:40:59.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 309237645314 2026-02-22T20:40:59.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:00.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:41:00.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:00.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645314 -lt 309237645314 2026-02-22T20:41:00.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:00.511 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-283467841540 2026-02-22T20:41:00.511 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:00.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:00.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-283467841540 2026-02-22T20:41:00.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:00.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=283467841540 2026-02-22T20:41:00.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 283467841540' 2026-02-22T20:41:00.515 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 283467841540 2026-02-22T20:41:00.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:00.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 283467841540 -lt 283467841540 2026-02-22T20:41:00.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:00.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:00.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:00.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:00.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:00.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:00.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:00.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:00.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:00.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:00.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:01.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:01.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:01.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:01.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:01.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:01.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:01.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:01.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:01.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ3 2026-02-22T20:41:01.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 3 % 2 2026-02-22T20:41:01.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-22T20:41:01.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1111: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:01.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:01.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:01.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:01.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:01.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2026-02-22T20:41:01.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:01.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:01.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:41:01.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:01.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:01.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2026-02-22T20:41:01.720 INFO:tasks.workunit.client.0.trial150.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2026-02-22T20:41:01.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:41:01.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:01.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:01.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:41:01.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:01.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:01.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:01.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:01.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:01.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:01.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:41:01.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:41:01.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:01.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:01.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:01.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:01.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:01.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:01.820 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:01.820 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:01.820 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:01.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:01.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:01.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:01.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:01.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:01.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:01.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:01.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:01.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:01.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:01.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:01.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:41:01.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:41:01.821 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:41:01.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:01.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:41:01.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:41:01.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:01.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:01.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:01.835 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:01.822+0000 7fd2cd563a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:01.836 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:01.826+0000 7fd2cd563a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:01.837 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:01.826+0000 7fd2cd563a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:01.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:41:01.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:01.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:41:01.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:01.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:01.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:01.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:01.983 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:01.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:01.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:02.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:02.222 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:02.210+0000 7fd2cd563a40 -1 Falling back to public interface 2026-02-22T20:41:02.831 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:02.818+0000 7fd2cd563a40 -1 osd.1 73 log_to_monitors true 2026-02-22T20:41:03.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:03.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:03.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:03.151 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:03.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:03.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:03.305 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 77 up_thru 77 down_at 74 last_clean_interval [66,73) [v2:127.0.0.1:6810/3929316398,v1:127.0.0.1:6811/3929316398] [v2:127.0.0.1:6812/3929316398,v1:127.0.0.1:6813/3929316398] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:41:03.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:03.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:03.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:03.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:03.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:03.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:03.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:03.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:03.306 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:03.306 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:03.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:03.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:03.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:03.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:03.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:03.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:03.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:03.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:03.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:03.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:03.590 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:03.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:03.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:03.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:03.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=309237645316 2026-02-22T20:41:03.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 309237645316 2026-02-22T20:41:03.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-309237645316' 2026-02-22T20:41:03.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:03.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:03.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481794 2026-02-22T20:41:03.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481794 2026-02-22T20:41:03.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-309237645316 1-330712481794' 2026-02-22T20:41:03.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:03.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-309237645316 2026-02-22T20:41:03.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:03.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:03.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-309237645316 2026-02-22T20:41:03.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:03.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=309237645316 2026-02-22T20:41:03.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 309237645316' 2026-02-22T20:41:03.739 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 309237645316 2026-02-22T20:41:03.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:03.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645316 -lt 309237645316 2026-02-22T20:41:03.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:03.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-330712481794 2026-02-22T20:41:03.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:03.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:03.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-330712481794 2026-02-22T20:41:03.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:03.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=330712481794 2026-02-22T20:41:03.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 330712481794' 2026-02-22T20:41:03.909 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 330712481794 2026-02-22T20:41:03.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:04.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 330712481794 2026-02-22T20:41:04.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:05.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:41:05.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:05.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 330712481794 2026-02-22T20:41:05.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:06.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:41:06.233 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:06.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481794 -lt 330712481794 2026-02-22T20:41:06.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:06.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:06.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:06.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:06.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:06.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:06.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:06.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:06.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:06.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:06.582 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:06.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:06.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:06.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:06.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:06.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:06.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:06.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:06.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:06.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ4 2026-02-22T20:41:06.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 4 % 2 2026-02-22T20:41:06.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-22T20:41:06.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:06.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1116: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-22T20:41:06.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:06.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:06.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:41:06.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:06.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-22T20:41:06.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:06.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:06.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:41:06.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:06.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:41:06.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:06.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:06.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:06.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:06.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:07.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:07.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-22T20:41:07.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:07.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:07.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:41:07.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:07.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:41:07.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-22T20:41:07.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:41:07.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:07.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:07.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:41:07.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:07.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:41:07.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:07.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:07.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:07.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:07.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:41:07.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:41:07.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:07.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:07.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:07.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:07.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:07.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:07.504 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:07.504 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:07.504 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:41:07.505 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:41:07.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:07.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:41:07.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:41:07.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:07.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:07.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:07.530 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:07.518+0000 7f02c87cca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:07.531 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:07.518+0000 7f02c87cca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:07.532 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:07.518+0000 7f02c87cca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:07.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:41:07.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:07.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:41:07.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:07.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:07.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:07.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:07.667 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:07.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:07.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:41:07.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:07.926 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:07.914+0000 7f02c87cca40 -1 Falling back to public interface 2026-02-22T20:41:08.485 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:08.474+0000 7f02c87cca40 -1 osd.0 78 log_to_monitors true 2026-02-22T20:41:08.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:08.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:08.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:08.828 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:08.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:08.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:41:08.988 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 82 up_thru 82 down_at 79 last_clean_interval [72,78) [v2:127.0.0.1:6802/1714363657,v1:127.0.0.1:6803/1714363657] [v2:127.0.0.1:6804/1714363657,v1:127.0.0.1:6805/1714363657] exists,up fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:41:08.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:08.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:08.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:08.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:08.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:08.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:08.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:08.989 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:08.990 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:08.990 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:08.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:08.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:08.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:09.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:09.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:09.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:09.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:09.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:09.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:09.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:09.263 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:09.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:09.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:09.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:09.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=352187318274 2026-02-22T20:41:09.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 352187318274 2026-02-22T20:41:09.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318274' 2026-02-22T20:41:09.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:09.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:09.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=330712481796 2026-02-22T20:41:09.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 330712481796 2026-02-22T20:41:09.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318274 1-330712481796' 2026-02-22T20:41:09.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:09.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-352187318274 2026-02-22T20:41:09.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:09.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:09.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-352187318274 2026-02-22T20:41:09.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:09.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=352187318274 2026-02-22T20:41:09.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 352187318274' 2026-02-22T20:41:09.415 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 352187318274 2026-02-22T20:41:09.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:09.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 309237645316 -lt 352187318274 2026-02-22T20:41:09.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:10.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:41:10.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:10.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 352187318274 -lt 352187318274 2026-02-22T20:41:10.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:10.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-330712481796 2026-02-22T20:41:10.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:10.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:10.749 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-330712481796 2026-02-22T20:41:10.749 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:10.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=330712481796 2026-02-22T20:41:10.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 330712481796' 2026-02-22T20:41:10.751 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 330712481796 2026-02-22T20:41:10.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:10.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 330712481796 -lt 330712481796 2026-02-22T20:41:10.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:10.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:10.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:11.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:11.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:11.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:11.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:11.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:11.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:11.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:11.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:11.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:11.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:11.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:11.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:11.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:11.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:11.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:11.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:11.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ5 2026-02-22T20:41:11.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 5 % 2 2026-02-22T20:41:11.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-22T20:41:11.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:11.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1121: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-22T20:41:11.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:11.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:11.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:41:11.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:11.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-22T20:41:11.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:11.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:11.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:41:11.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:11.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:41:11.459 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:11.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:11.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:11.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:11.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:11.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:11.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-22T20:41:11.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:11.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:11.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:41:11.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:11.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:11.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2026-02-22T20:41:12.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:41:12.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:12.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:12.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:12.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:12.059 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:12.059 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:12.059 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:12.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:12.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:12.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:12.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:12.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:12.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:12.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:12.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:12.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:12.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:12.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:12.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:41:12.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:41:12.060 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:41:12.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:12.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:41:12.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:41:12.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:12.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:12.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:12.082 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:12.070+0000 7fdc9386aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:12.083 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:12.070+0000 7fdc9386aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:12.084 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:12.074+0000 7fdc9386aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:12.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:41:12.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:12.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:41:12.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:12.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:12.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:12.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:12.218 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:12.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:12.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:12.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:12.474 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:12.462+0000 7fdc9386aa40 -1 Falling back to public interface 2026-02-22T20:41:13.042 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:13.030+0000 7fdc9386aa40 -1 osd.1 84 log_to_monitors true 2026-02-22T20:41:13.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:13.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:13.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:13.381 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:13.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:13.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:13.538 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 88 up_thru 88 down_at 85 last_clean_interval [77,84) [v2:127.0.0.1:6810/4024046899,v1:127.0.0.1:6811/4024046899] [v2:127.0.0.1:6812/4024046899,v1:127.0.0.1:6813/4024046899] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:41:13.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:13.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:13.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:13.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:13.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:13.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:13.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:13.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:13.540 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:13.540 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:13.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:13.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:13.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:13.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:13.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:13.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:13.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:13.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:13.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:13.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:13.817 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:13.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:13.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:13.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:13.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=352187318276 2026-02-22T20:41:13.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 352187318276 2026-02-22T20:41:13.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318276' 2026-02-22T20:41:13.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:13.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:13.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122050 2026-02-22T20:41:13.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122050 2026-02-22T20:41:13.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-352187318276 1-377957122050' 2026-02-22T20:41:13.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:13.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-352187318276 2026-02-22T20:41:13.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:13.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:13.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-352187318276 2026-02-22T20:41:13.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:13.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=352187318276 2026-02-22T20:41:13.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 352187318276' 2026-02-22T20:41:13.976 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 352187318276 2026-02-22T20:41:13.976 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:14.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 352187318275 -lt 352187318276 2026-02-22T20:41:14.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:15.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:41:15.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:15.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 352187318275 -lt 352187318276 2026-02-22T20:41:15.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:16.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:41:16.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:16.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 352187318276 -lt 352187318276 2026-02-22T20:41:16.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:16.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122050 2026-02-22T20:41:16.472 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:16.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:16.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122050 2026-02-22T20:41:16.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:16.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122050 2026-02-22T20:41:16.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122050' 2026-02-22T20:41:16.476 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 377957122050 2026-02-22T20:41:16.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:16.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122050 -lt 377957122050 2026-02-22T20:41:16.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:16.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:16.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:16.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:16.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:16.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:16.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:16.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:16.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:16.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:16.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:16.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:16.986 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:16.987 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:16.987 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:17.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:17.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:17.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:17.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:17.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ6 2026-02-22T20:41:17.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 6 % 2 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1126: TEST_corrupt_scrub_replicated: echo extra 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1127: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:17.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:41:17.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:17.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:17.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:17.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:17.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:17.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:17.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-22T20:41:17.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:17.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:17.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:41:17.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:17.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:41:17.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:17.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:17.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:17.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:17.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:17.754 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:17.754 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:17.754 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:17.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:17.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:17.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:17.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:17.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:17.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:17.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:17.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:17.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:17.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:17.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:17.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:41:17.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:41:17.756 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:41:17.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:17.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:41:17.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:41:17.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:17.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:17.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:17.776 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:17.762+0000 7fd4e1370a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:17.777 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:17.766+0000 7fd4e1370a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:17.778 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:17.766+0000 7fd4e1370a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:17.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:41:17.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:17.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:41:17.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:17.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:17.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:17.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:17.920 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:17.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:17.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:41:18.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:18.158 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:18.146+0000 7fd4e1370a40 -1 Falling back to public interface 2026-02-22T20:41:18.748 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:18.734+0000 7fd4e1370a40 -1 osd.0 89 log_to_monitors true 2026-02-22T20:41:19.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:19.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:19.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:19.081 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:19.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:19.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:41:19.243 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 93 up_thru 93 down_at 90 last_clean_interval [82,89) [v2:127.0.0.1:6802/3747506089,v1:127.0.0.1:6803/3747506089] [v2:127.0.0.1:6804/3747506089,v1:127.0.0.1:6805/3747506089] exists,up fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:41:19.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:19.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:19.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:19.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:19.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:19.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:19.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:19.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:19.244 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:19.244 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:19.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:19.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:19.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:19.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:19.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:19.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:19.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:19.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:19.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:19.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:19.509 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:19.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:19.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:19.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:19.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=399431958530 2026-02-22T20:41:19.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 399431958530 2026-02-22T20:41:19.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958530' 2026-02-22T20:41:19.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:19.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:19.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=377957122052 2026-02-22T20:41:19.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 377957122052 2026-02-22T20:41:19.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958530 1-377957122052' 2026-02-22T20:41:19.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:19.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-399431958530 2026-02-22T20:41:19.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:19.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:19.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-399431958530 2026-02-22T20:41:19.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:19.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=399431958530 2026-02-22T20:41:19.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 399431958530' 2026-02-22T20:41:19.661 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 399431958530 2026-02-22T20:41:19.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:19.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958530 -lt 399431958530 2026-02-22T20:41:19.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:19.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-377957122052 2026-02-22T20:41:19.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:19.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:19.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-377957122052 2026-02-22T20:41:19.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:19.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=377957122052 2026-02-22T20:41:19.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 377957122052' 2026-02-22T20:41:19.824 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 377957122052 2026-02-22T20:41:19.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:19.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 377957122052 -lt 377957122052 2026-02-22T20:41:19.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:19.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:19.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:20.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:20.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:20.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:20.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:20.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:20.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:20.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:20.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:20.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:20.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:20.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:20.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:20.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:20.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:20.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:20.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1128: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/extra-val 2026-02-22T20:41:20.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:20.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ7 2026-02-22T20:41:20.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 7 % 2 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1133: TEST_corrupt_scrub_replicated: echo -n newheader 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1134: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:41:20.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:20.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:20.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:20.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-22T20:41:20.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:20.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:20.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:41:20.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:20.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:20.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-22T20:41:21.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:41:21.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:21.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:21.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:21.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:21.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:21.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:41:21.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:41:21.165 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:41:21.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:21.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:41:21.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:41:21.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:21.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:21.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:21.179 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:21.166+0000 7fea8f6c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:21.181 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:21.170+0000 7fea8f6c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:21.182 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:21.170+0000 7fea8f6c7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:21.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:41:21.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:21.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:41:21.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:21.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:21.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:21.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:21.322 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:21.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:21.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:21.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:21.570 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:21.558+0000 7fea8f6c7a40 -1 Falling back to public interface 2026-02-22T20:41:22.163 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:22.150+0000 7fea8f6c7a40 -1 osd.1 95 log_to_monitors true 2026-02-22T20:41:22.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:22.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:22.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:22.487 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:22.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:22.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:22.644 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 99 up_thru 99 down_at 96 last_clean_interval [88,95) [v2:127.0.0.1:6810/3133090132,v1:127.0.0.1:6811/3133090132] [v2:127.0.0.1:6812/3133090132,v1:127.0.0.1:6813/3133090132] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:41:22.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:22.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:22.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:22.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:22.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:22.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:22.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:22.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:22.645 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:22.645 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:22.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:22.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:22.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:22.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:22.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:22.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:22.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:22.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:22.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:22.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:22.924 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:22.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:22.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:22.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:22.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=399431958531 2026-02-22T20:41:22.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 399431958531 2026-02-22T20:41:22.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958531' 2026-02-22T20:41:22.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:22.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:23.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=425201762306 2026-02-22T20:41:23.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 425201762306 2026-02-22T20:41:23.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-399431958531 1-425201762306' 2026-02-22T20:41:23.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:23.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-399431958531 2026-02-22T20:41:23.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:23.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:23.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-399431958531 2026-02-22T20:41:23.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:23.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=399431958531 2026-02-22T20:41:23.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 399431958531' 2026-02-22T20:41:23.077 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 399431958531 2026-02-22T20:41:23.078 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:23.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958530 -lt 399431958531 2026-02-22T20:41:23.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:24.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:41:24.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:24.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 399431958531 -lt 399431958531 2026-02-22T20:41:24.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:24.401 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-425201762306 2026-02-22T20:41:24.401 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:24.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:24.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-425201762306 2026-02-22T20:41:24.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:24.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=425201762306 2026-02-22T20:41:24.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 425201762306' 2026-02-22T20:41:24.405 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 425201762306 2026-02-22T20:41:24.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:24.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 425201762306 -lt 425201762306 2026-02-22T20:41:24.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:24.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:24.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:24.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:24.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:24.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:24.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:24.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:24.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:24.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:24.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:24.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:24.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:24.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:24.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:25.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:25.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:25.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:25.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1135: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/hdr 2026-02-22T20:41:25.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:25.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ8 2026-02-22T20:41:25.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 8 % 2 2026-02-22T20:41:25.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-22T20:41:25.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:25.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1139: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2026-02-22T20:41:25.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1140: TEST_corrupt_scrub_replicated: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2026-02-22T20:41:25.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1143: TEST_corrupt_scrub_replicated: echo -n bad-val 2026-02-22T20:41:25.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1144: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-22T20:41:25.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:25.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:25.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:41:25.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:25.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-22T20:41:25.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:25.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:25.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:41:25.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:25.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:41:25.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:25.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:25.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:25.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:25.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:25.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:25.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-22T20:41:25.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:25.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:25.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:41:25.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:25.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:41:25.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-22T20:41:25.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:41:25.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:25.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:25.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:41:25.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:25.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:41:25.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:25.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:25.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:25.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:25.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:41:25.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:41:25.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:25.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:25.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:25.721 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:25.721 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:25.721 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:25.721 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:25.721 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:25.722 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:25.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:25.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:25.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:25.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:25.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:25.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:25.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:25.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:25.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:25.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:25.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:25.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:41:25.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:41:25.724 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:41:25.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:25.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:41:25.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:41:25.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:25.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:25.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:25.737 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:25.726+0000 7fde09090a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:25.739 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:25.726+0000 7fde09090a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:25.740 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:25.726+0000 7fde09090a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:25.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:41:25.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:25.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:41:25.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:25.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:25.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:25.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:25.882 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:25.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:25.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:41:26.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:26.146 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:26.134+0000 7fde09090a40 -1 Falling back to public interface 2026-02-22T20:41:26.867 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:26.854+0000 7fde09090a40 -1 osd.0 101 log_to_monitors true 2026-02-22T20:41:27.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:27.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:27.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:27.050 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:27.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:27.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:41:27.205 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 105 up_thru 105 down_at 102 last_clean_interval [93,101) [v2:127.0.0.1:6802/1121551083,v1:127.0.0.1:6803/1121551083] [v2:127.0.0.1:6804/1121551083,v1:127.0.0.1:6805/1121551083] exists,up fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:41:27.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:27.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:27.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:27.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:27.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:27.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:27.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:27.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:27.206 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:27.206 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:27.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:27.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:27.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:27.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:27.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:27.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:27.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:27.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:27.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:27.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:27.483 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:27.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:27.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:27.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:27.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=450971566082 2026-02-22T20:41:27.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 450971566082 2026-02-22T20:41:27.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-450971566082' 2026-02-22T20:41:27.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:27.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:27.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=425201762308 2026-02-22T20:41:27.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 425201762308 2026-02-22T20:41:27.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-450971566082 1-425201762308' 2026-02-22T20:41:27.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:27.632 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-450971566082 2026-02-22T20:41:27.632 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:27.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:27.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-450971566082 2026-02-22T20:41:27.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:27.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=450971566082 2026-02-22T20:41:27.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 450971566082' 2026-02-22T20:41:27.635 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 450971566082 2026-02-22T20:41:27.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:27.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566082 -lt 450971566082 2026-02-22T20:41:27.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:27.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-425201762308 2026-02-22T20:41:27.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:27.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:27.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-425201762308 2026-02-22T20:41:27.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:27.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=425201762308 2026-02-22T20:41:27.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 425201762308' 2026-02-22T20:41:27.811 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 425201762308 2026-02-22T20:41:27.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:27.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 425201762308 -lt 425201762308 2026-02-22T20:41:27.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:27.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:27.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:28.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:28.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:28.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:28.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:28.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:28.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:28.167 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:28.167 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:28.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:28.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:28.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:28.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:28.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:28.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:28.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:28.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1145: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-22T20:41:28.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:28.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:28.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:41:28.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:28.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-22T20:41:28.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:28.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:28.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:41:28.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:28.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:41:28.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:28.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:28.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:28.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:28.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:28.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:28.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-22T20:41:28.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:28.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:28.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:41:28.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:28.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:28.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-22T20:41:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:41:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:41:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:29.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:29.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:41:29.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:41:29.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:29.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:29.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:29.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:29.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:29.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:29.117 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:29.117 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:29.118 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:29.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:29.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:29.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:29.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:29.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:29.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:29.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:29.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:29.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:29.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:29.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:29.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:41:29.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:41:29.120 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:41:29.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:29.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:41:29.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:41:29.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:29.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:29.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:29.136 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:29.122+0000 7fbaef9c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:29.137 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:29.126+0000 7fbaef9c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:29.138 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:29.126+0000 7fbaef9c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:29.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:41:29.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:29.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:41:29.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:29.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:29.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:29.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:29.282 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:29.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:29.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:29.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:29.514 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:29.502+0000 7fbaef9c4a40 -1 Falling back to public interface 2026-02-22T20:41:30.173 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:30.162+0000 7fbaef9c4a40 -1 osd.1 106 log_to_monitors true 2026-02-22T20:41:30.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:30.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:30.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:30.442 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:30.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:30.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:30.598 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 110 up_thru 110 down_at 107 last_clean_interval [99,106) [v2:127.0.0.1:6810/3649559897,v1:127.0.0.1:6811/3649559897] [v2:127.0.0.1:6812/3649559897,v1:127.0.0.1:6813/3649559897] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:41:30.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:30.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:30.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:30.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:30.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:30.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:30.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:30.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:30.600 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:30.600 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:30.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:30.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:30.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:30.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:30.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:30.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:30.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:30.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:30.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:30.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:30.875 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:30.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:30.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:30.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:30.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=450971566083 2026-02-22T20:41:30.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 450971566083 2026-02-22T20:41:30.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-450971566083' 2026-02-22T20:41:30.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:30.951 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:31.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=472446402562 2026-02-22T20:41:31.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 472446402562 2026-02-22T20:41:31.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-450971566083 1-472446402562' 2026-02-22T20:41:31.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:31.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-450971566083 2026-02-22T20:41:31.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:31.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:31.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-450971566083 2026-02-22T20:41:31.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:31.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=450971566083 2026-02-22T20:41:31.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 450971566083' 2026-02-22T20:41:31.029 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 450971566083 2026-02-22T20:41:31.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:31.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566082 -lt 450971566083 2026-02-22T20:41:31.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:32.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:41:32.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:32.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566083 -lt 450971566083 2026-02-22T20:41:32.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:32.352 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-472446402562 2026-02-22T20:41:32.352 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:32.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:32.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-472446402562 2026-02-22T20:41:32.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:32.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=472446402562 2026-02-22T20:41:32.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 472446402562' 2026-02-22T20:41:32.356 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 472446402562 2026-02-22T20:41:32.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:32.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 472446402562 -lt 472446402562 2026-02-22T20:41:32.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:32.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:32.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:32.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:32.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:32.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:32.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:32.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:32.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:32.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:32.703 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:32.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:32.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:32.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:32.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1146: TEST_corrupt_scrub_replicated: echo -n val3-ROBJ8 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1147: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:33.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:41:33.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:33.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:33.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:33.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:33.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:33.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:33.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-22T20:41:33.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:33.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:33.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:41:33.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:33.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:33.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-22T20:41:33.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:41:33.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:33.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:33.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:41:33.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:33.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:33.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:33.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:33.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:33.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:33.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:41:33.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:41:33.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:33.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:33.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:33.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:33.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:33.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:33.624 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:33.625 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:33.625 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:33.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:33.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:33.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:33.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:33.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:33.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:33.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:33.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:33.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:33.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:33.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:33.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:41:33.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:41:33.627 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:41:33.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:33.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:41:33.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:41:33.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:33.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:33.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:33.640 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:33.630+0000 7f1263215a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:33.642 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:33.630+0000 7f1263215a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:33.643 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:33.630+0000 7f1263215a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:33.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:41:33.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:33.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:41:33.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:33.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:33.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:33.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:33.788 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:33.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:33.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:33.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:34.022 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:34.010+0000 7f1263215a40 -1 Falling back to public interface 2026-02-22T20:41:34.572 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:34.558+0000 7f1263215a40 -1 osd.1 112 log_to_monitors true 2026-02-22T20:41:34.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:34.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:34.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:34.946 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:34.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:34.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:35.102 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 116 up_thru 116 down_at 113 last_clean_interval [110,112) [v2:127.0.0.1:6810/2335591957,v1:127.0.0.1:6811/2335591957] [v2:127.0.0.1:6812/2335591957,v1:127.0.0.1:6813/2335591957] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:41:35.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:35.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:35.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:35.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:35.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:35.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:35.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:35.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:35.103 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:35.103 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:35.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:35.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:35.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:35.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:35.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:35.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:35.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:35.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:35.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:35.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:35.374 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:35.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:35.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:35.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:35.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=450971566085 2026-02-22T20:41:35.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 450971566085 2026-02-22T20:41:35.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-450971566085' 2026-02-22T20:41:35.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:35.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:35.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=498216206338 2026-02-22T20:41:35.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 498216206338 2026-02-22T20:41:35.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-450971566085 1-498216206338' 2026-02-22T20:41:35.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:35.520 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-450971566085 2026-02-22T20:41:35.520 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:35.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:35.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-450971566085 2026-02-22T20:41:35.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:35.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=450971566085 2026-02-22T20:41:35.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 450971566085' 2026-02-22T20:41:35.524 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 450971566085 2026-02-22T20:41:35.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:35.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566084 -lt 450971566085 2026-02-22T20:41:35.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:36.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:41:36.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:36.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566085 -lt 450971566085 2026-02-22T20:41:36.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:36.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-498216206338 2026-02-22T20:41:36.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:36.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:36.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-498216206338 2026-02-22T20:41:36.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:36.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=498216206338 2026-02-22T20:41:36.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 498216206338' 2026-02-22T20:41:36.859 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 498216206338 2026-02-22T20:41:36.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:37.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 498216206338 -lt 498216206338 2026-02-22T20:41:37.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:37.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:37.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:37.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:37.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:37.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:37.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:37.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:37.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:37.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:37.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:37.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:37.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:37.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:37.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:37.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:37.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:37.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:37.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1148: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2026-02-22T20:41:37.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:37.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ9 2026-02-22T20:41:37.560 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 9 % 2 2026-02-22T20:41:37.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-22T20:41:37.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:37.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1152: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-22T20:41:37.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:37.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:37.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:41:37.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:37.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-22T20:41:37.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:37.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:37.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:41:37.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:37.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:41:37.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:37.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:37.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:37.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:37.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:37.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:37.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-22T20:41:37.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:37.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:37.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:41:37.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:37.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:37.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2026-02-22T20:41:38.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:41:38.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:38.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:38.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:41:38.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:38.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:38.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:38.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:38.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:38.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:38.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:41:38.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:41:38.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:38.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:38.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:38.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:38.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:38.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:38.033 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:38.033 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:38.033 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:38.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:38.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:38.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:38.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:38.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:38.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:38.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:38.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:38.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:38.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:38.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:38.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:41:38.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:41:38.035 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:41:38.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:38.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:41:38.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:41:38.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:38.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:38.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:38.055 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:38.042+0000 7ff1b5d74a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:38.056 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:38.046+0000 7ff1b5d74a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:38.057 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:38.046+0000 7ff1b5d74a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:38.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:41:38.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:38.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:41:38.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:38.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:38.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:38.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:38.193 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:38.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:38.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:38.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:38.422 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:38.410+0000 7ff1b5d74a40 -1 Falling back to public interface 2026-02-22T20:41:39.049 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:39.038+0000 7ff1b5d74a40 -1 osd.1 117 log_to_monitors true 2026-02-22T20:41:39.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:39.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:39.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:39.356 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:39.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:39.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:39.510 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 121 up_thru 121 down_at 118 last_clean_interval [116,117) [v2:127.0.0.1:6810/2292200801,v1:127.0.0.1:6811/2292200801] [v2:127.0.0.1:6812/2292200801,v1:127.0.0.1:6813/2292200801] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:41:39.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:39.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:39.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:39.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:39.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:39.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:39.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:39.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:39.512 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:39.512 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:39.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:39.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:39.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:39.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:39.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:39.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:39.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:39.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:39.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:39.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:39.787 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:39.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:39.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:39.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:39.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=450971566087 2026-02-22T20:41:39.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 450971566087 2026-02-22T20:41:39.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-450971566087' 2026-02-22T20:41:39.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:39.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:39.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=519691042818 2026-02-22T20:41:39.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 519691042818 2026-02-22T20:41:39.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-450971566087 1-519691042818' 2026-02-22T20:41:39.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:39.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-450971566087 2026-02-22T20:41:39.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:39.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:39.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-450971566087 2026-02-22T20:41:39.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:39.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=450971566087 2026-02-22T20:41:39.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 450971566087' 2026-02-22T20:41:39.936 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 450971566087 2026-02-22T20:41:39.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:40.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566086 -lt 450971566087 2026-02-22T20:41:40.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:41.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:41:41.106 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:41.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566086 -lt 450971566087 2026-02-22T20:41:41.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:42.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:41:42.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:42.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566087 -lt 450971566087 2026-02-22T20:41:42.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:42.434 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-519691042818 2026-02-22T20:41:42.434 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:42.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:42.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-519691042818 2026-02-22T20:41:42.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:42.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=519691042818 2026-02-22T20:41:42.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 519691042818' 2026-02-22T20:41:42.439 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 519691042818 2026-02-22T20:41:42.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:42.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 519691042818 -lt 519691042818 2026-02-22T20:41:42.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:42.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:42.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:42.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:42.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:42.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:42.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:42.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:42.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:42.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:42.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:42.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:42.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:42.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:42.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:43.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:43.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:43.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:43.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1153: TEST_corrupt_scrub_replicated: echo -n D 2026-02-22T20:41:43.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1154: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2026-02-22T20:41:43.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1155: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-22T20:41:43.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:43.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:43.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:41:43.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:43.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-22T20:41:43.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:43.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:43.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:41:43.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:43.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:41:43.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:43.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:43.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:43.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:43.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:43.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:43.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-22T20:41:43.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:43.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:43.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:41:43.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:43.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:43.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-22T20:41:43.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:41:43.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:43.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:41:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:41:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:41:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:43.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:43.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:43.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:43.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:43.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:43.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:43.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:43.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:41:43.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:41:43.774 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:41:43.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:43.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:41:43.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:41:43.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:43.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:43.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:43.794 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:43.782+0000 7fe6b0c33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:43.795 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:43.782+0000 7fe6b0c33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:43.796 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:43.786+0000 7fe6b0c33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:43.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:41:43.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:43.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:41:43.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:43.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:43.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:43.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:43.933 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:43.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:43.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:44.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:44.166 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:44.154+0000 7fe6b0c33a40 -1 Falling back to public interface 2026-02-22T20:41:44.707 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:44.694+0000 7fe6b0c33a40 -1 osd.1 123 log_to_monitors true 2026-02-22T20:41:45.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:45.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:45.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:45.102 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:45.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:45.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:45.258 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 127 up_thru 127 down_at 124 last_clean_interval [121,123) [v2:127.0.0.1:6810/4238385134,v1:127.0.0.1:6811/4238385134] [v2:127.0.0.1:6812/4238385134,v1:127.0.0.1:6813/4238385134] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:41:45.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:45.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:45.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:45.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:45.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:45.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:45.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:45.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:45.259 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:45.259 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:45.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:45.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:45.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:45.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:45.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:45.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:45.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:45.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:45.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:45.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:45.540 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:45.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:45.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:45.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:45.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=450971566089 2026-02-22T20:41:45.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 450971566089 2026-02-22T20:41:45.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-450971566089' 2026-02-22T20:41:45.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:45.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:45.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=545460846594 2026-02-22T20:41:45.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 545460846594 2026-02-22T20:41:45.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-450971566089 1-545460846594' 2026-02-22T20:41:45.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:45.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-450971566089 2026-02-22T20:41:45.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:45.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:45.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-450971566089 2026-02-22T20:41:45.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:45.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=450971566089 2026-02-22T20:41:45.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 450971566089' 2026-02-22T20:41:45.695 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 450971566089 2026-02-22T20:41:45.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:45.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566089 -lt 450971566089 2026-02-22T20:41:45.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:45.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-545460846594 2026-02-22T20:41:45.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:45.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:45.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-545460846594 2026-02-22T20:41:45.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:45.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=545460846594 2026-02-22T20:41:45.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 545460846594' 2026-02-22T20:41:45.857 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 545460846594 2026-02-22T20:41:45.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:46.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 545460846594 -lt 545460846594 2026-02-22T20:41:46.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:46.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:46.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:46.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:46.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:46.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:46.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:46.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:46.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:46.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:46.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:46.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:46.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:46.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:46.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:46.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:46.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:46.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:46.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1156: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2026-02-22T20:41:46.557 INFO:tasks.workunit.client.0.trial150.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2026-02-22T20:41:46.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:46.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ10 2026-02-22T20:41:46.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 10 % 2 2026-02-22T20:41:46.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-22T20:41:46.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:46.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:46.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ11 2026-02-22T20:41:46.560 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 11 % 2 2026-02-22T20:41:46.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-22T20:41:46.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:46.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:46.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ12 2026-02-22T20:41:46.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 12 % 2 2026-02-22T20:41:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-22T20:41:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:46.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ13 2026-02-22T20:41:46.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 13 % 2 2026-02-22T20:41:46.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-22T20:41:46.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:46.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:46.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ14 2026-02-22T20:41:46.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 14 % 2 2026-02-22T20:41:46.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-22T20:41:46.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:46.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1165: TEST_corrupt_scrub_replicated: echo -n bad-val 2026-02-22T20:41:46.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1166: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-22T20:41:46.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:46.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:46.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:41:46.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:46.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-22T20:41:46.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:46.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:46.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:41:46.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:46.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:41:46.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:46.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:46.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:46.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:46.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:46.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:46.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-22T20:41:46.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:46.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:46.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:41:46.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:46.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:41:46.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:47.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:47.150 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:47.150 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:47.150 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:47.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:47.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:41:47.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:41:47.152 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:41:47.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:47.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:41:47.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:41:47.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:47.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:47.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:47.172 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:47.158+0000 7fc584211a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:47.173 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:47.162+0000 7fc584211a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:47.174 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:47.162+0000 7fc584211a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:47.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:41:47.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:47.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:41:47.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:47.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:47.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:47.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:47.316 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:47.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:47.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:41:47.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:47.566 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:47.554+0000 7fc584211a40 -1 Falling back to public interface 2026-02-22T20:41:48.197 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:48.186+0000 7fc584211a40 -1 osd.0 129 log_to_monitors true 2026-02-22T20:41:48.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:48.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:48.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:48.474 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:48.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:48.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:41:48.630 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 133 up_thru 133 down_at 130 last_clean_interval [105,129) [v2:127.0.0.1:6802/427824878,v1:127.0.0.1:6803/427824878] [v2:127.0.0.1:6804/427824878,v1:127.0.0.1:6805/427824878] exists,up fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:41:48.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:48.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:48.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:48.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:48.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:48.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:48.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:48.631 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:48.632 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:48.632 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:48.632 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:48.632 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:48.632 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:48.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:48.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:48.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:48.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:48.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:48.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:48.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:48.899 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:48.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:48.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:48.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:48.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=571230650370 2026-02-22T20:41:48.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 571230650370 2026-02-22T20:41:48.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-571230650370' 2026-02-22T20:41:48.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:48.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:49.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=545460846595 2026-02-22T20:41:49.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 545460846595 2026-02-22T20:41:49.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-571230650370 1-545460846595' 2026-02-22T20:41:49.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:49.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-571230650370 2026-02-22T20:41:49.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:49.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:49.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-571230650370 2026-02-22T20:41:49.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:49.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=571230650370 2026-02-22T20:41:49.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 571230650370' 2026-02-22T20:41:49.056 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 571230650370 2026-02-22T20:41:49.057 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:49.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 450971566089 -lt 571230650370 2026-02-22T20:41:49.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:50.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:41:50.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:50.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 571230650370 -lt 571230650370 2026-02-22T20:41:50.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:50.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-545460846595 2026-02-22T20:41:50.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:50.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:50.395 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-545460846595 2026-02-22T20:41:50.395 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:50.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=545460846595 2026-02-22T20:41:50.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 545460846595' 2026-02-22T20:41:50.397 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 545460846595 2026-02-22T20:41:50.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:50.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 545460846595 -lt 545460846595 2026-02-22T20:41:50.556 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:50.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:50.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:50.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:50.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:50.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:50.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:50.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:50.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:50.758 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:50.758 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:50.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:50.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:50.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:50.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1167: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-22T20:41:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:41:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-22T20:41:51.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:51.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:51.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:41:51.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:51.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:41:51.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:51.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:51.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:51.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:51.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:51.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:51.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-22T20:41:51.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:51.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:51.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:41:51.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:51.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:51.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:51.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:51.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:51.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:51.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:51.698 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:51.698 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:51.698 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:51.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:51.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:51.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:51.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:51.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:51.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:51.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:51.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:51.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:51.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:51.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:51.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:41:51.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:41:51.700 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:41:51.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:51.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:41:51.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:41:51.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:51.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:51.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:51.717 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:51.706+0000 7fef63cefa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:51.718 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:51.706+0000 7fef63cefa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:51.719 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:51.706+0000 7fef63cefa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:41:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:41:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:51.863 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:51.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:52.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:52.094 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:52.082+0000 7fef63cefa40 -1 Falling back to public interface 2026-02-22T20:41:52.682 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:52.670+0000 7fef63cefa40 -1 osd.1 134 log_to_monitors true 2026-02-22T20:41:53.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:53.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:53.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:53.024 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:53.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:53.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:53.180 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 138 up_thru 138 down_at 135 last_clean_interval [127,134) [v2:127.0.0.1:6810/2185254021,v1:127.0.0.1:6811/2185254021] [v2:127.0.0.1:6812/2185254021,v1:127.0.0.1:6813/2185254021] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:41:53.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:53.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:53.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:53.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:53.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:53.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:53.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:53.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:53.181 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:53.181 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:53.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:53.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:53.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:53.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:53.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:53.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:53.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:53.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:53.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:53.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:53.459 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:53.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:53.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:53.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:53.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=571230650372 2026-02-22T20:41:53.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 571230650372 2026-02-22T20:41:53.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-571230650372' 2026-02-22T20:41:53.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:53.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:53.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=592705486850 2026-02-22T20:41:53.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 592705486850 2026-02-22T20:41:53.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-571230650372 1-592705486850' 2026-02-22T20:41:53.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:53.603 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-571230650372 2026-02-22T20:41:53.603 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:53.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:53.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-571230650372 2026-02-22T20:41:53.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:53.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=571230650372 2026-02-22T20:41:53.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 571230650372' 2026-02-22T20:41:53.608 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 571230650372 2026-02-22T20:41:53.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:53.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 571230650372 -lt 571230650372 2026-02-22T20:41:53.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:53.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-592705486850 2026-02-22T20:41:53.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:53.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:53.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-592705486850 2026-02-22T20:41:53.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:53.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=592705486850 2026-02-22T20:41:53.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 592705486850' 2026-02-22T20:41:53.780 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 592705486850 2026-02-22T20:41:53.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:53.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 592705486850 -lt 592705486850 2026-02-22T20:41:53.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:53.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:53.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:54.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:54.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:54.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:54.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:54.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:54.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:54.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:54.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:54.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:54.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:54.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:54.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:54.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:54.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:54.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:54.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1168: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/bad-val 2026-02-22T20:41:54.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:54.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ15 2026-02-22T20:41:54.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 15 % 2 2026-02-22T20:41:54.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-22T20:41:54.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:54.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1172: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-22T20:41:54.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:54.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:54.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:41:54.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:54.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-22T20:41:54.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:54.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:54.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:41:54.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:54.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:41:54.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:54.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:54.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:54.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:54.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:54.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:54.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-22T20:41:54.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:54.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:54.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:41:54.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:54.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:54.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2026-02-22T20:41:54.995 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:55.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:55.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:55.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:55.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:55.086 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:55.086 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:55.086 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:55.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:55.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:55.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:55.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:55.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:55.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:55.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:55.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:55.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:55.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:55.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:55.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:41:55.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:41:55.088 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:41:55.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:55.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:41:55.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:41:55.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:55.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:55.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:55.110 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:55.098+0000 7fa92a88ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:55.112 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:55.098+0000 7fa92a88ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:55.113 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:55.102+0000 7fa92a88ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:55.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:41:55.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:55.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:41:55.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:55.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:55.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:55.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:55.247 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:55.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:55.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:55.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:41:55.510 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:55.498+0000 7fa92a88ca40 -1 Falling back to public interface 2026-02-22T20:41:56.183 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:56.170+0000 7fa92a88ca40 -1 osd.1 140 log_to_monitors true 2026-02-22T20:41:56.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:41:56.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:56.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:41:56.414 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:41:56.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:56.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:41:56.577 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 144 up_thru 144 down_at 141 last_clean_interval [138,140) [v2:127.0.0.1:6810/1031233023,v1:127.0.0.1:6811/1031233023] [v2:127.0.0.1:6812/1031233023,v1:127.0.0.1:6813/1031233023] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:41:56.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:41:56.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:41:56.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:41:56.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:41:56.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:41:56.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:41:56.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:41:56.578 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:41:56.578 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:41:56.578 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:41:56.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:41:56.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:41:56.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:41:56.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:41:56.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:41:56.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:41:56.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:41:56.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:41:56.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:41:56.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:41:56.852 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:41:56.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:41:56.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:56.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:41:56.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=571230650373 2026-02-22T20:41:56.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 571230650373 2026-02-22T20:41:56.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-571230650373' 2026-02-22T20:41:56.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:41:56.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:41:57.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=618475290626 2026-02-22T20:41:57.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 618475290626 2026-02-22T20:41:57.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-571230650373 1-618475290626' 2026-02-22T20:41:57.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:57.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-571230650373 2026-02-22T20:41:57.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:57.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:41:57.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-571230650373 2026-02-22T20:41:57.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:57.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=571230650373 2026-02-22T20:41:57.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 571230650373' 2026-02-22T20:41:57.005 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 571230650373 2026-02-22T20:41:57.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:57.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 571230650372 -lt 571230650373 2026-02-22T20:41:57.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:41:58.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:41:58.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:41:58.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 571230650373 -lt 571230650373 2026-02-22T20:41:58.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:41:58.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-618475290626 2026-02-22T20:41:58.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:41:58.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:41:58.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-618475290626 2026-02-22T20:41:58.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:41:58.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=618475290626 2026-02-22T20:41:58.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2026-02-22T20:41:58.322 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 618475290626 2026-02-22T20:41:58.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:41:58.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 618475290626 -lt 618475290626 2026-02-22T20:41:58.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:41:58.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:58.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:58.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:41:58.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:41:58.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:41:58.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:41:58.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:41:58.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:41:58.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:41:58.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:41:58.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:41:58.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:41:58.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:41:58.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:41:59.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:41:59.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:41:59.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:41:59.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:41:59.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ16 2026-02-22T20:41:59.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 16 % 2 2026-02-22T20:41:59.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-22T20:41:59.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:41:59.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1176: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-22T20:41:59.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:41:59.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:41:59.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:41:59.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:41:59.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-22T20:41:59.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:41:59.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:41:59.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:41:59.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:41:59.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:41:59.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:41:59.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:41:59.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:41:59.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:41:59.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:41:59.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:41:59.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-22T20:41:59.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:41:59.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:41:59.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:41:59.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:41:59.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:41:59.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2026-02-22T20:41:59.538 INFO:tasks.workunit.client.0.trial150.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:41:59.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:41:59.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:41:59.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:41:59.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:41:59.650 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:41:59.650 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:41:59.650 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:41:59.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:59.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:41:59.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:41:59.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:41:59.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:41:59.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:41:59.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:41:59.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:41:59.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:41:59.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:41:59.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:41:59.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:41:59.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:41:59.652 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:41:59.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:41:59.652 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:41:59.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:41:59.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:41:59.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:41:59.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:41:59.672 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:59.658+0000 7fc28a550a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:59.673 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:59.662+0000 7fc28a550a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:59.674 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:41:59.662+0000 7fc28a550a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:41:59.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:41:59.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:41:59.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:41:59.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:41:59.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:41:59.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:41:59.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:41:59.812 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:41:59.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:41:59.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:41:59.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:42:00.062 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:00.050+0000 7fc28a550a40 -1 Falling back to public interface 2026-02-22T20:42:00.635 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:00.622+0000 7fc28a550a40 -1 osd.0 145 log_to_monitors true 2026-02-22T20:42:00.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:42:00.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:00.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:42:00.969 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:42:00.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:00.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:42:01.130 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 149 up_thru 149 down_at 146 last_clean_interval [133,145) [v2:127.0.0.1:6802/21136768,v1:127.0.0.1:6803/21136768] [v2:127.0.0.1:6804/21136768,v1:127.0.0.1:6805/21136768] exists,up fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:42:01.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:42:01.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:42:01.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:42:01.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:42:01.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:42:01.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:42:01.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:42:01.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:42:01.132 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:42:01.132 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:42:01.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:42:01.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:42:01.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:42:01.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:42:01.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:42:01.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:42:01.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:42:01.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:42:01.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:42:01.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:42:01.418 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:42:01.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:42:01.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:01.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:42:01.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=639950127106 2026-02-22T20:42:01.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 639950127106 2026-02-22T20:42:01.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127106' 2026-02-22T20:42:01.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:01.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:42:01.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=618475290628 2026-02-22T20:42:01.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 618475290628 2026-02-22T20:42:01.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127106 1-618475290628' 2026-02-22T20:42:01.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:01.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-639950127106 2026-02-22T20:42:01.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:01.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:42:01.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-639950127106 2026-02-22T20:42:01.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:01.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=639950127106 2026-02-22T20:42:01.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2026-02-22T20:42:01.571 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 639950127106 2026-02-22T20:42:01.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:01.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 571230650374 -lt 639950127106 2026-02-22T20:42:01.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:02.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:42:02.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:02.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127106 -lt 639950127106 2026-02-22T20:42:02.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:02.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-618475290628 2026-02-22T20:42:02.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:02.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:42:02.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-618475290628 2026-02-22T20:42:02.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:02.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=618475290628 2026-02-22T20:42:02.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 618475290628' 2026-02-22T20:42:02.904 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 618475290628 2026-02-22T20:42:02.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:03.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 618475290628 -lt 618475290628 2026-02-22T20:42:03.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:42:03.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:03.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:03.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:42:03.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:42:03.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:42:03.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:42:03.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:42:03.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:42:03.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:42:03.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:42:03.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:42:03.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:42:03.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:03.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:03.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:42:03.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:42:03.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:42:03.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1177: TEST_corrupt_scrub_replicated: echo -n bad-val 2026-02-22T20:42:03.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1178: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-22T20:42:03.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:42:03.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:42:03.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:42:03.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:42:03.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-22T20:42:03.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:42:03.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:42:03.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:42:03.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:42:03.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:42:03.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:42:03.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:42:03.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:42:03.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:42:03.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:42:03.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:42:03.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-22T20:42:03.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:42:03.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:42:03.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:42:03.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:42:03.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:42:03.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-22T20:42:04.093 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-22T20:42:04.093 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:42:04.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:42:04.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:42:04.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:42:04.186 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:42:04.186 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:04.186 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:04.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:42:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:42:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:42:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:42:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:42:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:42:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:42:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:42:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:42:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:42:04.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:42:04.188 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:42:04.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:42:04.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:42:04.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:42:04.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:42:04.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:42:04.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:42:04.208 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:04.194+0000 7f5ffec97a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:04.209 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:04.198+0000 7f5ffec97a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:04.210 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:04.198+0000 7f5ffec97a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:04.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:42:04.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:42:04.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:42:04.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:42:04.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:42:04.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:04.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:42:04.352 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:42:04.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:04.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:42:04.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:42:04.654 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:04.642+0000 7f5ffec97a40 -1 Falling back to public interface 2026-02-22T20:42:05.296 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:05.286+0000 7f5ffec97a40 -1 osd.1 150 log_to_monitors true 2026-02-22T20:42:05.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:42:05.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:05.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:42:05.512 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:42:05.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:05.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:42:05.675 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/3330393029,v1:127.0.0.1:6811/3330393029] [v2:127.0.0.1:6812/3330393029,v1:127.0.0.1:6813/3330393029] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:42:05.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:42:05.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:42:05.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:42:05.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:42:05.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:42:05.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:42:05.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:42:05.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:42:05.676 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:42:05.676 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:42:05.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:42:05.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:42:05.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:42:05.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:42:05.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:42:05.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:42:05.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:42:05.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:42:05.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:42:05.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:42:05.958 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:42:05.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:42:05.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:05.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:42:06.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=639950127108 2026-02-22T20:42:06.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 639950127108 2026-02-22T20:42:06.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127108' 2026-02-22T20:42:06.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:06.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:42:06.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=661424963586 2026-02-22T20:42:06.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 661424963586 2026-02-22T20:42:06.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127108 1-661424963586' 2026-02-22T20:42:06.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:06.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-639950127108 2026-02-22T20:42:06.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:06.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:42:06.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-639950127108 2026-02-22T20:42:06.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:06.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=639950127108 2026-02-22T20:42:06.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 639950127108' 2026-02-22T20:42:06.112 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 639950127108 2026-02-22T20:42:06.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:06.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127106 -lt 639950127108 2026-02-22T20:42:06.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:07.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:42:07.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:07.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127106 -lt 639950127108 2026-02-22T20:42:07.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:08.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:42:08.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:08.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127108 -lt 639950127108 2026-02-22T20:42:08.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:08.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-661424963586 2026-02-22T20:42:08.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:08.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:42:08.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-661424963586 2026-02-22T20:42:08.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:08.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=661424963586 2026-02-22T20:42:08.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2026-02-22T20:42:08.602 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 661424963586 2026-02-22T20:42:08.603 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:08.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 661424963586 -lt 661424963586 2026-02-22T20:42:08.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:42:08.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:08.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:08.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:42:08.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:42:08.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:42:08.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:42:08.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:42:08.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:42:08.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:42:08.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:42:09.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:42:09.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:42:09.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:09.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:09.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:42:09.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:42:09.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:42:09.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:42:09.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ17 2026-02-22T20:42:09.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 17 % 2 2026-02-22T20:42:09.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-22T20:42:09.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:42:09.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1183: TEST_corrupt_scrub_replicated: local payload=ROBJ17 2026-02-22T20:42:09.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1184: TEST_corrupt_scrub_replicated: echo ROBJ17 2026-02-22T20:42:09.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1185: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:42:09.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:42:09.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:42:09.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:42:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:42:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:42:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:42:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:42:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:42:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:42:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:42:09.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:42:09.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:42:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:42:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:42:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:42:09.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:42:09.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:42:09.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:42:09.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:42:09.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:42:09.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:42:09.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:42:09.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:42:09.799 INFO:tasks.workunit.client.0.trial150.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:42:09.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:42:09.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:42:09.902 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:42:09.902 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:09.902 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:09.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:09.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:09.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:42:09.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:42:09.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:42:09.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:42:09.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:42:09.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:42:09.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:42:09.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:42:09.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:42:09.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:42:09.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:42:09.904 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:42:09.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:42:09.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:42:09.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:42:09.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:42:09.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:42:09.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:42:09.923 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:09.910+0000 7fa57842da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:09.925 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:09.914+0000 7fa57842da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:09.926 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:09.914+0000 7fa57842da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:10.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:42:10.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:42:10.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:42:10.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:42:10.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:42:10.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:10.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:42:10.066 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:42:10.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:10.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:42:10.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:42:10.354 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:10.342+0000 7fa57842da40 -1 Falling back to public interface 2026-02-22T20:42:11.042 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:11.030+0000 7fa57842da40 -1 osd.0 156 log_to_monitors true 2026-02-22T20:42:11.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:42:11.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:11.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:42:11.226 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:42:11.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:11.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:42:11.379 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 160 up_thru 160 down_at 157 last_clean_interval [149,156) [v2:127.0.0.1:6802/1351061329,v1:127.0.0.1:6803/1351061329] [v2:127.0.0.1:6804/1351061329,v1:127.0.0.1:6805/1351061329] exists,up fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:42:11.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:42:11.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:42:11.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:42:11.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:42:11.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:42:11.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:42:11.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:42:11.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:42:11.380 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:42:11.381 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:42:11.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:42:11.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:42:11.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:42:11.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:42:11.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:42:11.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:42:11.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:42:11.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:42:11.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:42:11.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:42:11.657 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:42:11.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:42:11.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:11.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:42:11.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=687194767362 2026-02-22T20:42:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 687194767362 2026-02-22T20:42:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-687194767362' 2026-02-22T20:42:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:11.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:42:11.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=661424963588 2026-02-22T20:42:11.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 661424963588 2026-02-22T20:42:11.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-687194767362 1-661424963588' 2026-02-22T20:42:11.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:11.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-687194767362 2026-02-22T20:42:11.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:11.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:42:11.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-687194767362 2026-02-22T20:42:11.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:11.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=687194767362 2026-02-22T20:42:11.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 687194767362' 2026-02-22T20:42:11.808 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 687194767362 2026-02-22T20:42:11.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:11.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 687194767362 2026-02-22T20:42:11.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:12.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:42:12.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:13.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 687194767362 2026-02-22T20:42:13.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:14.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:42:14.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:14.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 687194767362 -lt 687194767362 2026-02-22T20:42:14.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:14.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-661424963588 2026-02-22T20:42:14.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:14.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:42:14.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-661424963588 2026-02-22T20:42:14.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:14.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=661424963588 2026-02-22T20:42:14.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 661424963588' 2026-02-22T20:42:14.309 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 661424963588 2026-02-22T20:42:14.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:14.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 661424963588 -lt 661424963588 2026-02-22T20:42:14.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:42:14.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:14.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:14.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:42:14.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:42:14.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:42:14.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:42:14.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:42:14.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:42:14.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:42:14.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:42:14.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:42:14.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:42:14.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:14.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1186: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:42:14.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:42:14.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:42:14.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:42:14.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:42:15.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:42:15.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:42:15.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:42:15.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:42:15.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:42:15.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:42:15.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:42:15.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:42:15.518 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-22T20:42:15.519 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-22T20:42:15.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:42:15.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:42:15.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:42:15.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:42:15.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:42:15.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:42:15.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:42:15.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:42:15.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:42:15.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:42:15.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:42:15.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:42:15.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:42:15.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:42:15.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:42:15.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:42:15.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:42:15.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:42:15.636 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:42:15.636 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:15.636 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:42:15.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:42:15.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:42:15.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:42:15.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:42:15.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:42:15.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:42:15.656 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:15.641+0000 7fa0e4f3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:15.657 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:15.645+0000 7fa0e4f3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:15.658 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:15.645+0000 7fa0e4f3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:15.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:42:15.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:42:15.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:42:15.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:42:15.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:42:15.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:15.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:42:15.800 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:42:15.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:15.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:42:15.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:42:15.994 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:15.981+0000 7fa0e4f3fa40 -1 Falling back to public interface 2026-02-22T20:42:16.609 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:16.593+0000 7fa0e4f3fa40 -1 osd.1 161 log_to_monitors true 2026-02-22T20:42:16.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:42:16.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:16.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:42:16.961 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:42:16.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:16.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:42:17.116 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 165 up_thru 165 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/2289690073,v1:127.0.0.1:6811/2289690073] [v2:127.0.0.1:6812/2289690073,v1:127.0.0.1:6813/2289690073] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:42:17.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:42:17.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:42:17.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:42:17.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:42:17.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:42:17.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:42:17.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:42:17.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:42:17.117 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:42:17.117 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:42:17.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:42:17.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:42:17.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:42:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:42:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:42:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:42:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:42:17.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:42:17.230 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:42:17.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:42:17.388 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:42:17.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:42:17.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:17.389 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:42:17.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=687194767364 2026-02-22T20:42:17.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 687194767364 2026-02-22T20:42:17.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-687194767364' 2026-02-22T20:42:17.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:17.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:42:17.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=708669603842 2026-02-22T20:42:17.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 708669603842 2026-02-22T20:42:17.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-687194767364 1-708669603842' 2026-02-22T20:42:17.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:17.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-687194767364 2026-02-22T20:42:17.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:17.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:42:17.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-687194767364 2026-02-22T20:42:17.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:17.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=687194767364 2026-02-22T20:42:17.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 687194767364' 2026-02-22T20:42:17.538 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 687194767364 2026-02-22T20:42:17.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:17.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 687194767362 -lt 687194767364 2026-02-22T20:42:17.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:18.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:42:18.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:18.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 687194767364 -lt 687194767364 2026-02-22T20:42:18.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:18.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-708669603842 2026-02-22T20:42:18.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:18.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:42:18.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-708669603842 2026-02-22T20:42:18.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:18.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=708669603842 2026-02-22T20:42:18.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2026-02-22T20:42:18.856 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 708669603842 2026-02-22T20:42:18.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:19.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 708669603842 -lt 708669603842 2026-02-22T20:42:19.019 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:42:19.019 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:19.019 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:19.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:42:19.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:42:19.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:42:19.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:42:19.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:42:19.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:42:19.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:42:19.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:42:19.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:42:19.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:42:19.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:19.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:19.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:42:19.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:42:19.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:42:19.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:42:19.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ18 2026-02-22T20:42:19.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 18 % 2 2026-02-22T20:42:19.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=0 2026-02-22T20:42:19.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:42:19.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1191: TEST_corrupt_scrub_replicated: local payload=ROBJ18 2026-02-22T20:42:19.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1192: TEST_corrupt_scrub_replicated: echo ROBJ18 2026-02-22T20:42:19.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1193: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:42:19.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:42:19.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:42:19.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:42:19.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:42:19.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:42:19.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:42:19.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:42:19.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:42:19.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:42:19.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:42:20.053 INFO:tasks.workunit.client.0.trial150.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-22T20:42:20.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:42:20.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:42:20.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:42:20.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:42:20.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:42:20.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:42:20.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:42:20.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:42:20.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:42:20.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:42:20.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:42:20.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:42:20.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:42:20.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:42:20.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:42:20.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:42:20.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:42:20.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:42:20.149 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:42:20.149 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:20.149 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:20.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:20.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:20.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:42:20.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:42:20.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:42:20.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:42:20.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:42:20.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:42:20.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:42:20.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:42:20.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:42:20.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:42:20.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:42:20.151 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:42:20.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:42:20.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:42:20.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:42:20.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:42:20.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:42:20.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:42:20.167 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:20.153+0000 7f6c6db0fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:20.169 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:20.157+0000 7f6c6db0fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:20.170 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:20.157+0000 7f6c6db0fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:20.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:42:20.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:42:20.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:42:20.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:42:20.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:42:20.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:20.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:42:20.314 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:42:20.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:20.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:42:20.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:42:20.602 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:20.589+0000 7f6c6db0fa40 -1 Falling back to public interface 2026-02-22T20:42:21.236 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:21.221+0000 7f6c6db0fa40 -1 osd.0 167 log_to_monitors true 2026-02-22T20:42:21.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:42:21.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:21.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:42:21.475 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:42:21.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:21.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:42:21.637 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 171 up_thru 171 down_at 168 last_clean_interval [160,167) [v2:127.0.0.1:6802/691076292,v1:127.0.0.1:6803/691076292] [v2:127.0.0.1:6804/691076292,v1:127.0.0.1:6805/691076292] exists,up fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:42:21.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:42:21.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:42:21.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:42:21.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:42:21.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:42:21.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:42:21.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:42:21.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:42:21.638 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:42:21.638 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:42:21.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:42:21.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:42:21.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:42:21.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:42:21.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:42:21.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:42:21.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:42:21.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:42:21.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:42:21.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:42:21.904 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:42:21.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:42:21.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:21.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:42:21.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407618 2026-02-22T20:42:21.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407618 2026-02-22T20:42:21.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407618' 2026-02-22T20:42:21.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:21.976 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:42:22.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=708669603844 2026-02-22T20:42:22.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 708669603844 2026-02-22T20:42:22.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407618 1-708669603844' 2026-02-22T20:42:22.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:22.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407618 2026-02-22T20:42:22.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:22.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:42:22.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407618 2026-02-22T20:42:22.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:22.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407618 2026-02-22T20:42:22.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407618' 2026-02-22T20:42:22.050 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 734439407618 2026-02-22T20:42:22.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:22.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 734439407618 2026-02-22T20:42:22.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:23.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:42:23.218 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:23.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 734439407618 2026-02-22T20:42:23.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:24.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:42:24.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:24.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407618 -lt 734439407618 2026-02-22T20:42:24.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:24.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-708669603844 2026-02-22T20:42:24.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:24.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:42:24.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-708669603844 2026-02-22T20:42:24.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:24.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=708669603844 2026-02-22T20:42:24.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 708669603844' 2026-02-22T20:42:24.536 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 708669603844 2026-02-22T20:42:24.537 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:24.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 708669603844 -lt 708669603844 2026-02-22T20:42:24.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:42:24.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:24.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:24.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:42:24.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:42:24.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:42:24.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:42:24.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:42:24.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:42:24.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:42:24.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:42:25.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:42:25.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:42:25.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:25.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:25.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:42:25.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:42:25.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:42:25.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1194: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:42:25.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:42:25.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:42:25.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:42:25.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:42:25.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:42:25.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:42:25.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:42:25.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:42:25.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:42:25.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:42:25.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:42:25.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:42:25.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:42:25.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:42:25.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:42:25.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:42:25.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:42:25.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:42:25.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:42:25.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:42:25.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:42:25.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:42:25.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:42:25.734 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-22T20:42:25.734 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-22T20:42:25.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:42:25.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:42:25.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:42:25.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:42:25.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:42:25.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:42:25.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:42:25.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:42:25.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:42:25.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:42:25.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:42:25.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:42:25.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:42:25.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:42:25.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:42:25.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:42:25.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:42:25.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:42:25.832 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:42:25.832 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:25.832 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:25.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:25.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:25.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:42:25.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:42:25.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:42:25.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:42:25.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:42:25.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:42:25.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:42:25.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:42:25.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:42:25.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:42:25.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:42:25.835 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:42:25.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:42:25.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:42:25.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:42:25.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:42:25.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:42:25.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:42:25.852 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:25.837+0000 7f5113a17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:25.853 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:25.841+0000 7f5113a17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:25.854 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:25.841+0000 7f5113a17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:25.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:42:25.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:42:25.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:42:25.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:42:25.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:42:25.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:25.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:42:25.993 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:42:25.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:25.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:42:26.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:42:26.250 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:26.237+0000 7f5113a17a40 -1 Falling back to public interface 2026-02-22T20:42:26.905 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:26.893+0000 7f5113a17a40 -1 osd.1 172 log_to_monitors true 2026-02-22T20:42:27.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:42:27.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:27.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:42:27.155 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:42:27.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:27.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:42:27.314 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 176 up_thru 176 down_at 173 last_clean_interval [165,172) [v2:127.0.0.1:6810/2807687097,v1:127.0.0.1:6811/2807687097] [v2:127.0.0.1:6812/2807687097,v1:127.0.0.1:6813/2807687097] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:42:27.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:42:27.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:42:27.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:42:27.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:42:27.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:42:27.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:42:27.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:42:27.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:42:27.315 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:42:27.315 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:42:27.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:42:27.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:42:27.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:42:27.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:42:27.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:42:27.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:42:27.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:42:27.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:42:27.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:42:27.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:42:27.592 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:42:27.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:42:27.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:27.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:42:27.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407620 2026-02-22T20:42:27.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407620 2026-02-22T20:42:27.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407620' 2026-02-22T20:42:27.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:27.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:42:27.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=755914244098 2026-02-22T20:42:27.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 755914244098 2026-02-22T20:42:27.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407620 1-755914244098' 2026-02-22T20:42:27.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:27.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407620 2026-02-22T20:42:27.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:27.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:42:27.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407620 2026-02-22T20:42:27.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:27.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407620 2026-02-22T20:42:27.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407620' 2026-02-22T20:42:27.740 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 734439407620 2026-02-22T20:42:27.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:27.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407620 -lt 734439407620 2026-02-22T20:42:27.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:27.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-755914244098 2026-02-22T20:42:27.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:27.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:42:27.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-755914244098 2026-02-22T20:42:27.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:27.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=755914244098 2026-02-22T20:42:27.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 755914244098' 2026-02-22T20:42:27.914 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 755914244098 2026-02-22T20:42:27.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:28.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 755914244098 2026-02-22T20:42:28.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:29.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:42:29.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:29.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 755914244098 2026-02-22T20:42:29.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:30.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:42:30.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:30.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 755914244098 -lt 755914244098 2026-02-22T20:42:30.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:42:30.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:30.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:30.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:42:30.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:42:30.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:42:30.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:42:30.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:42:30.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:42:30.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:42:30.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:42:30.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:42:30.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:42:30.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:30.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1196: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:42:30.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:42:30.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:42:30.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:42:30.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:42:30.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:42:31.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:42:31.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-22T20:42:31.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:42:31.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:42:31.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:42:31.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:42:31.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:42:31.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2026-02-22T20:42:31.444 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-22T20:42:31.444 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:42:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:42:31.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:42:31.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:42:31.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:42:31.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:42:31.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:42:31.534 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:42:31.534 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:31.534 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:31.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:42:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:42:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:42:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:42:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:42:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:42:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:42:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:42:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:42:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:42:31.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:42:31.536 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:42:31.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:42:31.537 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:42:31.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:42:31.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:42:31.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:42:31.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:42:31.556 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:31.541+0000 7feccf194a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:31.557 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:31.545+0000 7feccf194a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:31.558 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:31.545+0000 7feccf194a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:31.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:42:31.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:42:31.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:42:31.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:42:31.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:42:31.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:31.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:42:31.695 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:42:31.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:31.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:42:31.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:42:31.926 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:31.913+0000 7feccf194a40 -1 Falling back to public interface 2026-02-22T20:42:32.473 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:32.461+0000 7feccf194a40 -1 osd.1 177 log_to_monitors true 2026-02-22T20:42:32.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:42:32.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:32.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:42:32.862 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:42:32.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:32.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:42:33.016 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 181 up_thru 181 down_at 178 last_clean_interval [176,177) [v2:127.0.0.1:6810/415122340,v1:127.0.0.1:6811/415122340] [v2:127.0.0.1:6812/415122340,v1:127.0.0.1:6813/415122340] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:42:33.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:42:33.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:42:33.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:42:33.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:42:33.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:42:33.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:42:33.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:42:33.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:42:33.018 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:42:33.018 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:42:33.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:42:33.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:42:33.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:42:33.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:42:33.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:42:33.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:42:33.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:42:33.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:42:33.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:42:33.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:42:33.289 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:42:33.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:42:33.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:33.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:42:33.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407622 2026-02-22T20:42:33.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407622 2026-02-22T20:42:33.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407622' 2026-02-22T20:42:33.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:33.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:42:33.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080578 2026-02-22T20:42:33.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080578 2026-02-22T20:42:33.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407622 1-777389080578' 2026-02-22T20:42:33.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:33.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407622 2026-02-22T20:42:33.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:33.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:42:33.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407622 2026-02-22T20:42:33.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:33.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407622 2026-02-22T20:42:33.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407622' 2026-02-22T20:42:33.439 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 734439407622 2026-02-22T20:42:33.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:33.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407621 -lt 734439407622 2026-02-22T20:42:33.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:34.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:42:34.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:34.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407622 -lt 734439407622 2026-02-22T20:42:34.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:34.767 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080578 2026-02-22T20:42:34.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:34.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:42:34.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080578 2026-02-22T20:42:34.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:34.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080578 2026-02-22T20:42:34.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080578' 2026-02-22T20:42:34.771 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 777389080578 2026-02-22T20:42:34.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:34.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080578 -lt 777389080578 2026-02-22T20:42:34.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:42:34.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:34.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:35.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:42:35.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:42:35.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:42:35.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:42:35.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:42:35.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:42:35.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:42:35.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:42:35.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:42:35.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:42:35.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:35.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:35.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:42:35.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:42:35.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:42:35.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1088: TEST_corrupt_scrub_replicated: for i in $(seq 1 $total_objs) 2026-02-22T20:42:35.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1089: TEST_corrupt_scrub_replicated: objname=ROBJ19 2026-02-22T20:42:35.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: expr 19 % 2 2026-02-22T20:42:35.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1092: TEST_corrupt_scrub_replicated: local osd=1 2026-02-22T20:42:35.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1094: TEST_corrupt_scrub_replicated: case $i in 2026-02-22T20:42:35.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1205: TEST_corrupt_scrub_replicated: get_pg csr_pool ROBJ0 2026-02-22T20:42:35.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-22T20:42:35.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-22T20:42:35.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-22T20:42:35.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:42:35.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1205: TEST_corrupt_scrub_replicated: local pg=3.0 2026-02-22T20:42:35.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1207: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2026-02-22T20:42:35.705 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: {} 2026-02-22T20:42:35.706 INFO:tasks.workunit.client.0.trial150.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2026-02-22T20:42:35.714 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: {} 2026-02-22T20:42:35.714 INFO:tasks.workunit.client.0.trial150.stderr:osd.1: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_max_object_size = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2026-02-22T20:42:35.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1209: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2026-02-22T20:42:35.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-22T20:42:35.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-22T20:42:35.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-22T20:42:35.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-22T20:42:35.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-22T20:42:35.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-22T20:42:35.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ11 2026-02-22T20:42:35.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-22T20:42:35.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-22T20:42:35.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-22T20:42:35.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-22T20:42:35.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-22T20:42:35.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ11 2026-02-22T20:42:35.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-22T20:42:35.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2026-02-22T20:42:35.725 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2026-02-22T20:42:35.725 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:42:35.886 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:42:35.886 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:42:35.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-22T20:42:35.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-22T20:42:35.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-22T20:42:35.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-22T20:42:35.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-22T20:42:35.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-22T20:42:35.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-22T20:42:35.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-22T20:42:35.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-22T20:42:35.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:42:35.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-22T20:42:35.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T20:42:35.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:42:35.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:42:35.889 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-22T20:42:35.890 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:42:35.890 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:42:35.890 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:42:35.890 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:35.890 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:35.890 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:42:35.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-22T20:42:35.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:42:35.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-22T20:42:35.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-22T20:42:35.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-22T20:42:35.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:42:35.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:42:35.954 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:42:35.954 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:35.954 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:35.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:42:35.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-22T20:42:35.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.40073/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2026-02-22T20:42:36.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1210: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2026-02-22T20:42:36.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-22T20:42:36.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-22T20:42:36.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-22T20:42:36.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-22T20:42:36.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-22T20:42:36.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-22T20:42:36.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ12 2026-02-22T20:42:36.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-22T20:42:36.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-22T20:42:36.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-22T20:42:36.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-22T20:42:36.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-22T20:42:36.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ12 2026-02-22T20:42:36.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-22T20:42:36.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2026-02-22T20:42:36.015 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2026-02-22T20:42:36.016 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:42:36.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:42:36.179 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:42:36.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-22T20:42:36.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-22T20:42:36.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-22T20:42:36.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-22T20:42:36.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-22T20:42:36.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-22T20:42:36.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-22T20:42:36.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-22T20:42:36.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-22T20:42:36.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:42:36.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-22T20:42:36.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T20:42:36.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:42:36.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:42:36.185 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-22T20:42:36.185 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:42:36.185 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:42:36.185 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:42:36.186 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:36.186 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:36.186 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:42:36.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-22T20:42:36.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:42:36.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-22T20:42:36.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-22T20:42:36.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-22T20:42:36.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:42:36.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:42:36.244 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:42:36.244 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:36.244 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:36.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:42:36.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-22T20:42:36.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.40073/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2026-02-22T20:42:36.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1211: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2026-02-22T20:42:36.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-22T20:42:36.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-22T20:42:36.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-22T20:42:36.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-22T20:42:36.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-22T20:42:36.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-22T20:42:36.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-22T20:42:36.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-22T20:42:36.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-22T20:42:36.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-22T20:42:36.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-22T20:42:36.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-22T20:42:36.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-22T20:42:36.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-22T20:42:36.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-22T20:42:36.306 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-22T20:42:36.307 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:42:36.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:42:36.467 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:42:36.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-22T20:42:36.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-22T20:42:36.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-22T20:42:36.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-22T20:42:36.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-22T20:42:36.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-22T20:42:36.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-22T20:42:36.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-22T20:42:36.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-22T20:42:36.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:42:36.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-22T20:42:36.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T20:42:36.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:42:36.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:42:36.469 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-22T20:42:36.470 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:42:36.470 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:42:36.470 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:42:36.470 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:36.470 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:36.470 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:42:36.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-22T20:42:36.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:42:36.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-22T20:42:36.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-22T20:42:36.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-22T20:42:36.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:42:36.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:42:36.531 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:42:36.531 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:36.531 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:36.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:42:36.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-22T20:42:36.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.40073/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1212: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-22T20:42:36.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-22T20:42:36.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-22T20:42:36.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-22T20:42:36.592 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-22T20:42:36.592 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:42:36.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:42:36.757 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:42:36.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-22T20:42:36.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-22T20:42:36.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-22T20:42:36.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-22T20:42:36.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-22T20:42:36.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-22T20:42:36.758 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-22T20:42:36.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-22T20:42:36.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-22T20:42:36.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:42:36.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-22T20:42:36.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T20:42:36.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:42:36.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:42:36.761 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-22T20:42:36.761 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:42:36.761 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:42:36.761 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:42:36.761 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:36.761 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:36.761 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:42:36.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-22T20:42:36.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:42:36.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-22T20:42:36.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-22T20:42:36.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-22T20:42:36.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:42:36.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:42:36.822 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:42:36.822 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:36.822 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:36.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:42:36.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-22T20:42:36.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.40073/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2026-02-22T20:42:36.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1214: TEST_corrupt_scrub_replicated: pg_scrub 3.0 2026-02-22T20:42:36.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-22T20:42:36.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-22T20:42:36.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:42:36.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:42:36.881 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:42:36.881 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:42:36.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:42:36.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:42:36.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:42:37.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:42:37.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:42:37.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:42:37.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:42:37.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:42:37.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:42:37.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:42:37.340 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:42:37.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:42:37.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:37.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:42:37.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407624 2026-02-22T20:42:37.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407624 2026-02-22T20:42:37.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407624' 2026-02-22T20:42:37.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:37.412 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:42:37.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080579 2026-02-22T20:42:37.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080579 2026-02-22T20:42:37.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407624 1-777389080579' 2026-02-22T20:42:37.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:37.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407624 2026-02-22T20:42:37.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:37.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:42:37.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407624 2026-02-22T20:42:37.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:37.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407624 2026-02-22T20:42:37.490 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 734439407624 2026-02-22T20:42:37.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407624' 2026-02-22T20:42:37.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:37.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407622 -lt 734439407624 2026-02-22T20:42:37.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:38.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:42:38.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:38.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407624 -lt 734439407624 2026-02-22T20:42:38.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:38.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080579 2026-02-22T20:42:38.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:38.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:42:38.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080579 2026-02-22T20:42:38.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:38.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080579 2026-02-22T20:42:38.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080579' 2026-02-22T20:42:38.815 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 777389080579 2026-02-22T20:42:38.816 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:38.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080580 -lt 777389080579 2026-02-22T20:42:38.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:42:38.982 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:42:38.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:42:38.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:42:38.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:42:38.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:42:38.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:42:38.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:42:39.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T20:42:39.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:42:39.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:42:39.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:42:39.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-22T20:42:39.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:42:39.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:42:39.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:42:39.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:42:39.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-22T20:40:45.395897+0000 2026-02-22T20:42:39.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-22T20:42:39.360 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-22T20:42:39.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-22T20:40:45.395897+0000 2026-02-22T20:42:39.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:42:39.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:40:45.395897+0000 2026-02-22T20:42:39.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:42:39.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:42:39.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:42:39.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:42:39.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:42:39.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:42:39.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:42:39.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:42:39.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:45.395897+0000 '>' 2026-02-22T20:40:45.395897+0000 2026-02-22T20:42:39.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:42:40.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:42:40.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:42:40.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:42:40.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:42:40.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:42:40.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:42:40.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:42:40.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:45.395897+0000 '>' 2026-02-22T20:40:45.395897+0000 2026-02-22T20:42:40.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:42:41.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:42:41.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:42:41.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:42:41.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:42:41.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:42:41.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:42:41.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:42:41.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:45.395897+0000 '>' 2026-02-22T20:40:45.395897+0000 2026-02-22T20:42:41.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:42:42.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:42:42.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:42:42.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:42:42.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:42:42.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:42:42.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:42:42.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:42:43.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:45.395897+0000 '>' 2026-02-22T20:40:45.395897+0000 2026-02-22T20:42:43.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:42:44.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:42:44.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:42:44.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:42:44.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:42:44.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:42:44.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:42:44.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:42:39.377518+0000 '>' 2026-02-22T20:40:45.395897+0000 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1216: TEST_corrupt_scrub_replicated: ERRORS=0 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1217: TEST_corrupt_scrub_replicated: declare -a s_err_strings 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1218: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1219: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1220: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1221: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1222: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1223: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1224: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1225: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1226: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2026-02-22T20:42:44.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1227: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1228: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1229: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1230: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1231: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1232: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1233: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1234: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1235: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1236: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1237: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1238: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1239: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : candidate size 1 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049713/1049720 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 1 missing, 8 inconsistent objects' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 18 errors' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1241: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:42:44.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1243: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:123a5f55:::ROBJ19:head : size 1049600 > 1048576 is too large' td/osd-scrub-repair/osd.1.log 2026-02-22T20:42:44.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1250: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2026-02-22T20:42:44.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1252: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2026-02-22T20:42:44.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1252: TEST_corrupt_scrub_replicated: test 1 = 1 2026-02-22T20:42:44.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1254: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-22T20:42:44.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1254: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2026-02-22T20:42:44.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1256: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2026-02-22T20:42:44.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1258: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2026-02-22T20:42:44.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1258: TEST_corrupt_scrub_replicated: epoch=181 2026-02-22T20:42:44.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1260: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2026-02-22T20:42:44.419 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:42:44.419 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:42:44.419 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:42:44.419 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:42:44.419 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:42:44.419 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:42:44.419 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:42:44.420 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:42:44.420 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:42:44.420 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:42:44.420 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:42:44.420 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:42:44.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1260: TEST_corrupt_scrub_replicated: jq .inconsistents 2026-02-22T20:42:44.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1260: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:42:44.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2026-02-22T20:42:44.439 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:42:44.439 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:42:44.439 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:42:44.439 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:42:44.439 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:42:44.439 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:42:44.440 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:42:44.440 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:42:44.440 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:42:44.440 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:42:44.440 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:42:44.440 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-22T20:42:44.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: jq .inconsistents 2026-02-22T20:42:44.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2173: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:42:44.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2174: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:42:44.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:42:44.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2175: TEST_corrupt_scrub_replicated: test no = yes 2026-02-22T20:42:44.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2180: TEST_corrupt_scrub_replicated: test '' = yes 2026-02-22T20:42:44.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2185: TEST_corrupt_scrub_replicated: objname=ROBJ9 2026-02-22T20:42:44.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2187: TEST_corrupt_scrub_replicated: echo -n ZZZ 2026-02-22T20:42:44.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2188: TEST_corrupt_scrub_replicated: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2026-02-22T20:42:44.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2190: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-22T20:42:44.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:42:44.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:42:44.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:42:44.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:42:44.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-22T20:42:44.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:42:44.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:42:44.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:42:44.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:42:44.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:42:44.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:42:44.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:42:44.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:42:44.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:42:44.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:42:44.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:42:44.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-22T20:42:44.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:42:44.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:42:44.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:42:44.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:42:44.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:42:44.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-22T20:42:44.972 INFO:tasks.workunit.client.0.trial150.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:42:45.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:42:45.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:42:45.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:42:45.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:42:45.062 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:42:45.062 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:45.062 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:45.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:45.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:45.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:42:45.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:42:45.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:42:45.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:42:45.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:42:45.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:42:45.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:42:45.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:42:45.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:42:45.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:42:45.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:42:45.064 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:42:45.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:42:45.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:42:45.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:42:45.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:42:45.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:42:45.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:42:45.081 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:45.069+0000 7ff579b78a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:45.083 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:45.069+0000 7ff579b78a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:45.083 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:45.069+0000 7ff579b78a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:45.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:42:45.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:42:45.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:42:45.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:42:45.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:42:45.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:45.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:42:45.227 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:42:45.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:45.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:42:45.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:42:45.474 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:45.461+0000 7ff579b78a40 -1 Falling back to public interface 2026-02-22T20:42:46.104 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:46.089+0000 7ff579b78a40 -1 osd.0 183 log_to_monitors true 2026-02-22T20:42:46.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:42:46.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:46.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:42:46.391 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:42:46.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:46.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:42:46.547 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 187 up_thru 187 down_at 184 last_clean_interval [171,183) [v2:127.0.0.1:6802/593921839,v1:127.0.0.1:6803/593921839] [v2:127.0.0.1:6804/593921839,v1:127.0.0.1:6805/593921839] exists,up fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:42:46.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:42:46.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:42:46.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:42:46.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:42:46.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:42:46.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:42:46.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:42:46.548 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:42:46.549 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:42:46.549 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:42:46.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:42:46.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:42:46.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:42:46.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:42:46.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:42:46.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:42:46.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:42:46.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:42:46.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:42:46.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:42:46.824 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:42:46.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:42:46.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:46.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:42:46.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=803158884354 2026-02-22T20:42:46.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 803158884354 2026-02-22T20:42:46.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-803158884354' 2026-02-22T20:42:46.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:46.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:42:46.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080582 2026-02-22T20:42:46.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080582 2026-02-22T20:42:46.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-803158884354 1-777389080582' 2026-02-22T20:42:46.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:46.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-803158884354 2026-02-22T20:42:46.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:46.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:42:46.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-803158884354 2026-02-22T20:42:46.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:46.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=803158884354 2026-02-22T20:42:46.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 803158884354' 2026-02-22T20:42:46.973 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 803158884354 2026-02-22T20:42:46.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:47.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407625 -lt 803158884354 2026-02-22T20:42:47.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:48.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:42:48.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:48.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 803158884354 -lt 803158884354 2026-02-22T20:42:48.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:48.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080582 2026-02-22T20:42:48.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:48.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:42:48.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080582 2026-02-22T20:42:48.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:48.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080582 2026-02-22T20:42:48.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080582' 2026-02-22T20:42:48.308 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 777389080582 2026-02-22T20:42:48.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:48.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080583 -lt 777389080582 2026-02-22T20:42:48.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:42:48.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:48.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:48.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:42:48.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:42:48.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:42:48.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:42:48.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:42:48.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:42:48.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:42:48.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:42:48.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:42:48.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:42:48.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:48.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:49.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:42:49.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:42:49.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:42:49.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2191: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2026-02-22T20:42:49.030 INFO:tasks.workunit.client.0.trial150.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2026-02-22T20:42:49.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2193: TEST_corrupt_scrub_replicated: objname=ROBJ10 2026-02-22T20:42:49.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2194: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 get-attr _ 2026-02-22T20:42:49.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:42:49.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:42:49.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:42:49.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:42:49.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 get-attr _ 2026-02-22T20:42:49.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:42:49.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:42:49.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:42:49.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:42:49.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:42:49.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:42:49.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:42:49.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:42:49.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:42:49.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:42:49.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:42:49.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 get-attr _ 2026-02-22T20:42:49.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:42:49.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:42:49.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:42:49.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:42:49.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:42:49.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 get-attr _ 2026-02-22T20:42:49.404 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-22T20:42:49.404 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:42:49.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:42:49.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:42:49.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:42:49.498 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:42:49.498 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:49.498 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:49.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:49.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:49.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:42:49.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:42:49.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:42:49.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:42:49.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:42:49.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:42:49.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:42:49.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:42:49.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:42:49.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:42:49.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:42:49.500 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:42:49.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:42:49.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:42:49.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:42:49.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:42:49.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:42:49.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:42:49.523 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:49.509+0000 7fd34e5e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:49.525 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:49.513+0000 7fd34e5e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:49.526 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:49.513+0000 7fd34e5e3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:49.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:42:49.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:42:49.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:42:49.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:42:49.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:42:49.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:49.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:42:49.660 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:42:49.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:49.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:42:49.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:42:49.890 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:49.877+0000 7fd34e5e3a40 -1 Falling back to public interface 2026-02-22T20:42:50.495 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:50.481+0000 7fd34e5e3a40 -1 osd.1 189 log_to_monitors true 2026-02-22T20:42:50.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:42:50.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:50.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:42:50.827 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:42:50.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:50.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:42:50.990 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 192 up_thru 192 down_at 189 last_clean_interval [181,189) [v2:127.0.0.1:6810/99765628,v1:127.0.0.1:6811/99765628] [v2:127.0.0.1:6812/99765628,v1:127.0.0.1:6813/99765628] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:42:50.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:42:50.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:42:50.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:42:50.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:42:50.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:42:50.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:42:50.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:42:50.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:42:50.992 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:42:50.992 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:42:50.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:42:50.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:42:50.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:42:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:42:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:42:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:42:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:42:51.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:42:51.107 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:42:51.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:42:51.261 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:42:51.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:42:51.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:51.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:42:51.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=803158884356 2026-02-22T20:42:51.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 803158884356 2026-02-22T20:42:51.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-803158884356' 2026-02-22T20:42:51.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:51.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:42:51.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=824633720834 2026-02-22T20:42:51.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 824633720834 2026-02-22T20:42:51.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-803158884356 1-824633720834' 2026-02-22T20:42:51.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:51.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-803158884356 2026-02-22T20:42:51.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:51.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:42:51.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-803158884356 2026-02-22T20:42:51.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:51.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=803158884356 2026-02-22T20:42:51.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 803158884356' 2026-02-22T20:42:51.412 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 803158884356 2026-02-22T20:42:51.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:51.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 803158884354 -lt 803158884356 2026-02-22T20:42:51.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:52.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:42:52.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:52.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 803158884356 -lt 803158884356 2026-02-22T20:42:52.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:52.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-824633720834 2026-02-22T20:42:52.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:52.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:42:52.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-824633720834 2026-02-22T20:42:52.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:52.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=824633720834 2026-02-22T20:42:52.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 824633720834' 2026-02-22T20:42:52.748 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 824633720834 2026-02-22T20:42:52.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:52.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 824633720834 -lt 824633720834 2026-02-22T20:42:52.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:42:52.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:52.923 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:53.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:42:53.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:42:53.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:42:53.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:42:53.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:42:53.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:42:53.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:42:53.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:42:53.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:42:53.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:42:53.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:53.282 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:53.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:42:53.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:42:53.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:42:53.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2195: TEST_corrupt_scrub_replicated: rados --pool csr_pool setomapval ROBJ10 key2-ROBJ10 val2-ROBJ10 2026-02-22T20:42:53.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2196: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-22T20:42:53.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:42:53.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:42:53.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:42:53.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:42:53.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-22T20:42:53.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:42:53.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:42:53.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:42:53.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:42:53.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:42:53.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:42:53.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:42:53.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:42:53.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:42:53.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:42:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:42:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-22T20:42:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:42:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:42:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:42:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:42:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:42:53.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-22T20:42:54.001 INFO:tasks.workunit.client.0.trial150.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-22T20:42:54.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:42:54.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:42:54.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:42:54.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:42:54.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:42:54.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:42:54.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:42:54.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:42:54.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:54.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:54.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:54.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:42:54.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:42:54.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:42:54.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:42:54.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:42:54.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:42:54.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:42:54.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:42:54.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:42:54.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:42:54.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:42:54.112 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:42:54.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:42:54.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:42:54.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:42:54.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:42:54.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:42:54.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:42:54.128 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:54.113+0000 7f42e0d51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:54.129 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:54.117+0000 7f42e0d51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:54.130 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:54.117+0000 7f42e0d51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:54.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:42:54.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:42:54.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:42:54.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:42:54.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:42:54.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:54.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:42:54.272 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:42:54.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:54.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:42:54.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:42:54.507 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:54.493+0000 7f42e0d51a40 -1 Falling back to public interface 2026-02-22T20:42:55.118 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:55.105+0000 7f42e0d51a40 -1 osd.0 194 log_to_monitors true 2026-02-22T20:42:55.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:42:55.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:55.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:42:55.433 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:42:55.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:55.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:42:55.593 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 198 up_thru 198 down_at 195 last_clean_interval [187,194) [v2:127.0.0.1:6802/2137355346,v1:127.0.0.1:6803/2137355346] [v2:127.0.0.1:6804/2137355346,v1:127.0.0.1:6805/2137355346] exists,up fd75825d-4113-41d3-8c2e-0ab21035c87d 2026-02-22T20:42:55.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:42:55.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:42:55.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:42:55.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:42:55.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:42:55.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:42:55.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:42:55.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:42:55.595 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:42:55.596 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:42:55.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:42:55.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:42:55.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:42:55.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:42:55.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:42:55.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:42:55.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:42:55.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:42:55.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:42:55.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:42:55.867 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:42:55.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:42:55.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:55.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:42:55.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=850403524610 2026-02-22T20:42:55.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 850403524610 2026-02-22T20:42:55.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524610' 2026-02-22T20:42:55.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:42:55.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:42:56.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=824633720836 2026-02-22T20:42:56.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 824633720836 2026-02-22T20:42:56.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524610 1-824633720836' 2026-02-22T20:42:56.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:56.013 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-850403524610 2026-02-22T20:42:56.013 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:56.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:42:56.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-850403524610 2026-02-22T20:42:56.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:56.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=850403524610 2026-02-22T20:42:56.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 850403524610' 2026-02-22T20:42:56.017 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 850403524610 2026-02-22T20:42:56.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:56.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 850403524610 2026-02-22T20:42:56.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:57.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:42:57.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:57.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 850403524610 2026-02-22T20:42:57.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:42:58.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:42:58.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:42:58.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524610 -lt 850403524610 2026-02-22T20:42:58.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:42:58.508 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-824633720836 2026-02-22T20:42:58.508 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:42:58.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:42:58.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-824633720836 2026-02-22T20:42:58.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:42:58.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=824633720836 2026-02-22T20:42:58.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 824633720836' 2026-02-22T20:42:58.512 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 824633720836 2026-02-22T20:42:58.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:42:58.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 824633720836 -lt 824633720836 2026-02-22T20:42:58.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:42:58.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:58.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:58.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:42:58.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:42:58.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:42:58.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:42:58.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:42:58.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:42:58.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:42:58.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:42:59.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:42:59.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:42:59.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:42:59.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2197: TEST_corrupt_scrub_replicated: objectstore_tool td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:42:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:42:59.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:42:59.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:42:59.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:42:59.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:42:59.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:42:59.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:42:59.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-22T20:42:59.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:42:59.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:42:59.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:42:59.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:42:59.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:42:59.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ10 set-attr _ td/osd-scrub-repair/oi 2026-02-22T20:42:59.717 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-22T20:42:59.717 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-22T20:42:59.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:42:59.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:42:59.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:42:59.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:42:59.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:42:59.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:42:59.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:42:59.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:42:59.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:42:59.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:42:59.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:42:59.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:42:59.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:42:59.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:42:59.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:42:59.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:42:59.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:42:59.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:42:59.839 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:42:59.839 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:42:59.839 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:42:59.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:59.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:42:59.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:42:59.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:42:59.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:42:59.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:42:59.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:42:59.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:42:59.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:42:59.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:42:59.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:42:59.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:42:59.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:42:59.841 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:42:59.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:42:59.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:42:59.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:42:59.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:42:59.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:42:59.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:42:59.848 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:59.833+0000 7f086d5baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:59.850 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:59.837+0000 7f086d5baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:59.851 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:42:59.837+0000 7f086d5baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:42:59.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:42:59.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:42:59.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:42:59.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:42:59.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:42:59.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:42:59.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:42:59.990 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:42:59.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:42:59.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:43:00.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:43:00.226 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:00.213+0000 7f086d5baa40 -1 Falling back to public interface 2026-02-22T20:43:00.780 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:00.765+0000 7f086d5baa40 -1 osd.1 199 log_to_monitors true 2026-02-22T20:43:01.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:43:01.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:43:01.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:43:01.152 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:43:01.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:43:01.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:43:01.310 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 203 up_thru 203 down_at 200 last_clean_interval [192,199) [v2:127.0.0.1:6810/1793937233,v1:127.0.0.1:6811/1793937233] [v2:127.0.0.1:6812/1793937233,v1:127.0.0.1:6813/1793937233] exists,up 9384a6a4-a814-4e77-8a87-47e2d6def999 2026-02-22T20:43:01.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:43:01.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:43:01.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:43:01.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:43:01.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:43:01.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:43:01.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:43:01.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:43:01.311 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:43:01.311 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:43:01.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:43:01.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:43:01.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:43:01.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:43:01.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:43:01.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:43:01.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:43:01.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:43:01.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:43:01.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:43:01.587 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:43:01.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:43:01.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:01.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:43:01.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=850403524612 2026-02-22T20:43:01.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 850403524612 2026-02-22T20:43:01.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524612' 2026-02-22T20:43:01.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:01.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:43:01.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=871878361090 2026-02-22T20:43:01.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 871878361090 2026-02-22T20:43:01.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524612 1-871878361090' 2026-02-22T20:43:01.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:01.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-850403524612 2026-02-22T20:43:01.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:01.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:43:01.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-850403524612 2026-02-22T20:43:01.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:01.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=850403524612 2026-02-22T20:43:01.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 850403524612' 2026-02-22T20:43:01.736 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 850403524612 2026-02-22T20:43:01.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:01.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524612 -lt 850403524612 2026-02-22T20:43:01.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:01.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-871878361090 2026-02-22T20:43:01.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:01.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:43:01.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-871878361090 2026-02-22T20:43:01.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:01.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=871878361090 2026-02-22T20:43:01.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 871878361090' 2026-02-22T20:43:01.908 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 871878361090 2026-02-22T20:43:01.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:43:02.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 871878361090 2026-02-22T20:43:02.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:43:03.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:43:03.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:43:03.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 871878361090 2026-02-22T20:43:03.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:43:04.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:43:04.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:43:04.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 871878361090 -lt 871878361090 2026-02-22T20:43:04.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:43:04.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:43:04.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:43:04.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:43:04.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:43:04.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:43:04.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:43:04.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:43:04.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:43:04.590 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:43:04.590 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:43:04.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:43:04.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:43:04.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:43:04.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:43:04.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:43:04.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:43:04.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:43:04.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2198: TEST_corrupt_scrub_replicated: rm td/osd-scrub-repair/oi 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2200: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ11 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ11 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-22T20:43:04.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2026-02-22T20:43:04.940 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2026-02-22T20:43:04.940 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:43:05.100 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:43:05.100 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:43:05.100 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-22T20:43:05.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-22T20:43:05.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-22T20:43:05.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-22T20:43:05.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-22T20:43:05.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-22T20:43:05.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-22T20:43:05.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-22T20:43:05.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-22T20:43:05.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:43:05.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-22T20:43:05.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T20:43:05.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:43:05.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:43:05.104 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-22T20:43:05.104 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:43:05.104 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:43:05.104 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:43:05.105 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:05.105 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:05.105 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:43:05.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-22T20:43:05.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:43:05.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-22T20:43:05.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-22T20:43:05.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-22T20:43:05.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:43:05.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:43:05.169 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:43:05.169 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:05.169 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:05.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:43:05.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-22T20:43:05.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.40073/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2201: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ12 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-22T20:43:05.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-22T20:43:05.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ12 2026-02-22T20:43:05.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-22T20:43:05.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2026-02-22T20:43:05.231 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2026-02-22T20:43:05.231 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:43:05.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:43:05.393 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:43:05.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-22T20:43:05.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-22T20:43:05.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-22T20:43:05.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-22T20:43:05.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-22T20:43:05.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-22T20:43:05.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-22T20:43:05.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-22T20:43:05.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-22T20:43:05.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:43:05.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-22T20:43:05.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T20:43:05.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:43:05.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:43:05.396 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-22T20:43:05.396 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:43:05.396 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:43:05.397 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:43:05.397 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:05.397 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:05.397 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:43:05.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-22T20:43:05.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:43:05.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-22T20:43:05.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-22T20:43:05.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-22T20:43:05.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:43:05.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:43:05.461 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:43:05.462 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:05.462 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:05.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:43:05.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-22T20:43:05.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.40073/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2202: TEST_corrupt_scrub_replicated: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-22T20:43:05.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-22T20:43:05.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-22T20:43:05.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-22T20:43:05.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-22T20:43:05.523 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-22T20:43:05.523 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:43:05.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:43:05.684 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:43:05.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-22T20:43:05.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-22T20:43:05.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-22T20:43:05.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-22T20:43:05.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-22T20:43:05.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-22T20:43:05.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-22T20:43:05.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-22T20:43:05.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-22T20:43:05.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:43:05.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-22T20:43:05.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T20:43:05.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:43:05.686 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:43:05.687 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-22T20:43:05.687 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:43:05.687 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:43:05.687 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:43:05.687 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:05.688 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:05.688 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:43:05.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-22T20:43:05.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:43:05.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-22T20:43:05.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-22T20:43:05.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-22T20:43:05.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:43:05.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:43:05.751 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:43:05.751 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:05.752 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:05.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:43:05.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-22T20:43:05.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.40073/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2026-02-22T20:43:05.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2203: TEST_corrupt_scrub_replicated: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2026-02-22T20:43:05.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-22T20:43:05.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-22T20:43:05.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-22T20:43:05.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-22T20:43:05.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-22T20:43:05.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-22T20:43:05.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-22T20:43:05.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-22T20:43:05.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-22T20:43:05.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-22T20:43:05.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-22T20:43:05.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-22T20:43:05.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-22T20:43:05.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-22T20:43:05.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-22T20:43:05.811 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-22T20:43:05.811 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:43:05.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:43:05.971 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:43:05.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-22T20:43:05.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-22T20:43:05.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-22T20:43:05.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-22T20:43:05.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-22T20:43:05.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-22T20:43:05.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-22T20:43:05.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-22T20:43:05.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-22T20:43:05.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:43:05.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-22T20:43:05.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T20:43:05.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:43:05.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:43:05.975 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-22T20:43:05.975 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:43:05.975 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:43:05.975 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:43:05.975 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:05.975 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:05.975 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:43:05.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-22T20:43:06.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:43:06.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-22T20:43:06.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-22T20:43:06.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-22T20:43:06.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:43:06.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:43:06.038 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:43:06.038 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:06.038 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:06.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:43:06.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-22T20:43:06.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.40073/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2026-02-22T20:43:06.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2206: TEST_corrupt_scrub_replicated: ceph tell 'osd.*' injectargs -- --osd-max-object-size=134217728 2026-02-22T20:43:06.165 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: {} 2026-02-22T20:43:06.165 INFO:tasks.workunit.client.0.trial150.stderr:osd.0: osd_max_object_size = '' 2026-02-22T20:43:06.175 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: {} 2026-02-22T20:43:06.175 INFO:tasks.workunit.client.0.trial150.stderr:osd.1: osd_max_object_size = '' 2026-02-22T20:43:06.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2208: TEST_corrupt_scrub_replicated: pg_deep_scrub 3.0 2026-02-22T20:43:06.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-22T20:43:06.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-22T20:43:06.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:43:06.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:43:06.183 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:43:06.183 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:43:06.184 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:43:06.184 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:43:06.184 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:43:06.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:43:06.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:43:06.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:43:06.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:43:06.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:43:06.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:43:06.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:43:06.633 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:43:06.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:43:06.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:06.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:43:06.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=850403524614 2026-02-22T20:43:06.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 850403524614 2026-02-22T20:43:06.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524614' 2026-02-22T20:43:06.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:06.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:43:06.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=871878361092 2026-02-22T20:43:06.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 871878361092 2026-02-22T20:43:06.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524614 1-871878361092' 2026-02-22T20:43:06.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:06.782 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-850403524614 2026-02-22T20:43:06.782 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:06.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:43:06.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-850403524614 2026-02-22T20:43:06.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:06.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=850403524614 2026-02-22T20:43:06.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 850403524614' 2026-02-22T20:43:06.786 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 850403524614 2026-02-22T20:43:06.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:06.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524613 -lt 850403524614 2026-02-22T20:43:06.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:43:07.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:43:07.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:08.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524614 -lt 850403524614 2026-02-22T20:43:08.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:08.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-871878361092 2026-02-22T20:43:08.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:08.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:43:08.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-871878361092 2026-02-22T20:43:08.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:08.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=871878361092 2026-02-22T20:43:08.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 871878361092' 2026-02-22T20:43:08.118 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 871878361092 2026-02-22T20:43:08.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:43:08.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 871878361092 -lt 871878361092 2026-02-22T20:43:08.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:43:08.282 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:43:08.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:43:08.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:43:08.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:43:08.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:43:08.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:43:08.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:43:08.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T20:43:08.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:43:08.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:43:08.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:43:08.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:43:08.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:08.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:43:08.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:08.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:43:08.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-22T20:40:45.395897+0000 2026-02-22T20:43:08.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-22T20:43:08.665 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-22T20:43:08.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-22T20:40:45.395897+0000 last_deep_scrub_stamp 2026-02-22T20:43:08.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:43:08.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:40:45.395897+0000 2026-02-22T20:43:08.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-22T20:43:08.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:43:08.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:08.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:43:08.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:08.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:43:08.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:08.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:43:08.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:45.395897+0000 '>' 2026-02-22T20:40:45.395897+0000 2026-02-22T20:43:08.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:43:09.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:43:09.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:09.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:43:09.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:09.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:43:09.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:09.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:43:10.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:45.395897+0000 '>' 2026-02-22T20:40:45.395897+0000 2026-02-22T20:43:10.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:43:11.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:43:11.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:11.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:43:11.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:11.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:43:11.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:11.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:43:11.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:40:45.395897+0000 '>' 2026-02-22T20:40:45.395897+0000 2026-02-22T20:43:11.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:43:12.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:43:12.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:12.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:43:12.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:12.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:43:12.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:12.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:43:12.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:43:08.695032+0000 '>' 2026-02-22T20:40:45.395897+0000 2026-02-22T20:43:12.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2210: TEST_corrupt_scrub_replicated: err_strings=() 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2211: TEST_corrupt_scrub_replicated: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2212: TEST_corrupt_scrub_replicated: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2213: TEST_corrupt_scrub_replicated: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2214: TEST_corrupt_scrub_replicated: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2215: TEST_corrupt_scrub_replicated: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2216: TEST_corrupt_scrub_replicated: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2217: TEST_corrupt_scrub_replicated: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2218: TEST_corrupt_scrub_replicated: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2219: TEST_corrupt_scrub_replicated: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2220: TEST_corrupt_scrub_replicated: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2221: TEST_corrupt_scrub_replicated: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2222: TEST_corrupt_scrub_replicated: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' 2026-02-22T20:43:12.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2223: TEST_corrupt_scrub_replicated: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2224: TEST_corrupt_scrub_replicated: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2225: TEST_corrupt_scrub_replicated: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2226: TEST_corrupt_scrub_replicated: err_strings[15]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2227: TEST_corrupt_scrub_replicated: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2228: TEST_corrupt_scrub_replicated: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2229: TEST_corrupt_scrub_replicated: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2230: TEST_corrupt_scrub_replicated: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2231: TEST_corrupt_scrub_replicated: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2232: TEST_corrupt_scrub_replicated: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2233: TEST_corrupt_scrub_replicated: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2234: TEST_corrupt_scrub_replicated: err_strings[23]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2235: TEST_corrupt_scrub_replicated: err_strings[24]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2236: TEST_corrupt_scrub_replicated: err_strings[25]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' 2026-02-22T20:43:12.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2237: TEST_corrupt_scrub_replicated: err_strings[26]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2238: TEST_corrupt_scrub_replicated: err_strings[27]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2239: TEST_corrupt_scrub_replicated: err_strings[28]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2240: TEST_corrupt_scrub_replicated: err_strings[29]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2241: TEST_corrupt_scrub_replicated: err_strings[30]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2242: TEST_corrupt_scrub_replicated: err_strings[31]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2243: TEST_corrupt_scrub_replicated: err_strings[32]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2244: TEST_corrupt_scrub_replicated: err_strings[33]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2245: TEST_corrupt_scrub_replicated: err_strings[34]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2246: TEST_corrupt_scrub_replicated: err_strings[35]='log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2247: TEST_corrupt_scrub_replicated: err_strings[36]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2248: TEST_corrupt_scrub_replicated: err_strings[37]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2249: TEST_corrupt_scrub_replicated: err_strings[38]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:30259878:::ROBJ15:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)], object info inconsistent ' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:33aca486:::ROBJ18:head : data_digest 0xbd89c912 != data_digest 0x2ddbf8f5 from auth oi 3:33aca486:::ROBJ18:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 54 dd 2ddbf8f5 od ddc3680f alloc_hint [[]0 0 255[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:33aca486:::ROBJ18:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:5c7b2c47:::ROBJ16:head : candidate had a corrupt snapset' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:5c7b2c47:::ROBJ16:head : candidate had a missing snapset key' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:5c7b2c47:::ROBJ16:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:86586531:::ROBJ8:head : attr value mismatch '\''_key1-ROBJ8'\'', attr name mismatch '\''_key3-ROBJ8'\'', attr name mismatch '\''_key2-ROBJ8'\''' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:87abbf36:::ROBJ11:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8aa5320e:::ROBJ17:head : data_digest 0x5af0c3ef != data_digest 0x2ddbf8f5 from auth oi 3:8aa5320e:::ROBJ17:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 51 dd 2ddbf8f5 od e9572720 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8aa5320e:::ROBJ17:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0xefced57a != omap_digest 0x6a73cc07 from shard 1' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:8b55fa4b:::ROBJ7:head : omap_digest 0x6a73cc07 != omap_digest 0xefced57a from auth oi 3:8b55fa4b:::ROBJ7:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 21 dd 2ddbf8f5 od efced57a alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:a53c12e8:::ROBJ6:head : omap_digest 0x689ee887 != omap_digest 0x179c919f from shard 1, omap_digest 0x689ee887 != omap_digest 0x179c919f from auth oi 3:a53c12e8:::ROBJ6:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 18 dd 2ddbf8f5 od 179c919f alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:b1f19cbd:::ROBJ10:head : omap_digest 0xa8dd5adc != omap_digest 0xc2025a24 from auth oi 3:b1f19cbd:::ROBJ10:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 30 dd 2ddbf8f5 od c2025a24 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:b1f19cbd:::ROBJ10:head : failed to pick suitable auth object' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:bc819597:::ROBJ12:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:c0c86b1d:::ROBJ14:head : candidate had a missing info key' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:c0c86b1d:::ROBJ14:head : candidate had a corrupt info' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:c0c86b1d:::ROBJ14:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : candidate size 9 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:ce3f1d6a:::ROBJ1:head : data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from shard 0, data_digest 0x2d4a11c2 != data_digest 0x2ddbf8f5 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from auth oi 3:ce3f1d6a:::ROBJ1:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 3 dd 2ddbf8f5 od f5fba2c6 alloc_hint [[]0 0 0[]][)], size 9 != size 7 from shard 0' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:d60617f9:::ROBJ13:head : candidate had a read error' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:d60617f9:::ROBJ13:head : candidate had a stat error' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:d60617f9:::ROBJ13:head : failed to pick suitable object info' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:e97ce31e:::ROBJ2:head : data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from shard 1, data_digest 0x578a4830 != data_digest 0x2ddbf8f5 from auth oi 3:e97ce31e:::ROBJ2:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 6 dd 2ddbf8f5 od f8e11918 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 3:f2a5b2a4:::ROBJ3:head : missing' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:f4981d31:::ROBJ4:head : omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from shard 1, omap_digest 0xd7178dfe != omap_digest 0xe2d46ea4 from auth oi 3:f4981d31:::ROBJ4:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 12 dd 2ddbf8f5 od e2d46ea4 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x1a862a41 != omap_digest 0x6cac8f6 from shard 1' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 1 soid 3:f4bfd4d1:::ROBJ5:head : omap_digest 0x6cac8f6 != omap_digest 0x1a862a41 from auth oi 3:f4bfd4d1:::ROBJ5:head[(][0-9]*'\''[0-9]* osd.1.0:[0-9]* dirty|omap|data_digest|omap_digest s 7 uv 15 dd 2ddbf8f5 od 1a862a41 alloc_hint [[]0 0 0[]][)]' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : candidate size 3 info size 7 mismatch' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard 0 soid 3:ffdb2004:::ROBJ9:head : object info inconsistent ' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:c0c86b1d:::ROBJ14:head : no '\''_'\'' attr' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : deep-scrub [0-9]*[.]0 3:5c7b2c47:::ROBJ16:head : can'\''t decode '\''snapset'\'' attr .* v=3 cannot decode .* Malformed input' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub : stat mismatch, got 19/19 objects, 0/0 clones, 18/19 dirty, 18/19 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 1049715/1049716 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 1 missing, 11 inconsistent objects' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2251: TEST_corrupt_scrub_replicated: for err_string in "${err_strings[@]}" 2026-02-22T20:43:12.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2253: TEST_corrupt_scrub_replicated: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 deep-scrub 35 errors' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:12.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2260: TEST_corrupt_scrub_replicated: rados list-inconsistent-pg csr_pool 2026-02-22T20:43:12.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2262: TEST_corrupt_scrub_replicated: jq '. | length' td/osd-scrub-repair/json 2026-02-22T20:43:12.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2262: TEST_corrupt_scrub_replicated: test 1 = 1 2026-02-22T20:43:12.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2264: TEST_corrupt_scrub_replicated: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-22T20:43:12.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2264: TEST_corrupt_scrub_replicated: test 3.0 = 3.0 2026-02-22T20:43:12.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2266: TEST_corrupt_scrub_replicated: rados list-inconsistent-obj 3.0 2026-02-22T20:43:12.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2268: TEST_corrupt_scrub_replicated: jq .epoch td/osd-scrub-repair/json 2026-02-22T20:43:12.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2268: TEST_corrupt_scrub_replicated: epoch=203 2026-02-22T20:43:12.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2270: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2026-02-22T20:43:12.712 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2270: TEST_corrupt_scrub_replicated: jq .inconsistents 2026-02-22T20:43:12.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:2270: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:43:12.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3698: TEST_corrupt_scrub_replicated: jq 'def walk(f): 2026-02-22T20:43:12.736 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:43:12.736 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:43:12.736 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:43:12.736 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:43:12.736 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:43:12.737 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:43:12.737 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:43:12.737 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:43:12.737 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:43:12.737 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:43:12.737 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:43:12.737 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-22T20:43:12.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3698: TEST_corrupt_scrub_replicated: jq .inconsistents 2026-02-22T20:43:12.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3698: TEST_corrupt_scrub_replicated: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:43:12.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3699: TEST_corrupt_scrub_replicated: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:43:12.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:43:12.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3700: TEST_corrupt_scrub_replicated: test no = yes 2026-02-22T20:43:12.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3705: TEST_corrupt_scrub_replicated: test '' = yes 2026-02-22T20:43:12.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3710: TEST_corrupt_scrub_replicated: repair 3.0 2026-02-22T20:43:12.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=3.0 2026-02-22T20:43:12.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 3.0 2026-02-22T20:43:12.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:12.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:43:12.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:12.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:43:12.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:43:08.695032+0000 2026-02-22T20:43:12.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 3.0 2026-02-22T20:43:13.068 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0 on osd.1 to repair 2026-02-22T20:43:13.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 3.0 2026-02-22T20:43:08.695032+0000 2026-02-22T20:43:13.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:43:13.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:43:08.695032+0000 2026-02-22T20:43:13.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:43:13.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:43:13.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:13.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:43:13.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:13.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:43:13.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:13.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:43:13.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:43:08.695032+0000 '>' 2026-02-22T20:43:08.695032+0000 2026-02-22T20:43:13.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:43:14.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:43:14.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:14.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:43:14.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:14.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:43:14.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:14.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:43:14.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:43:08.695032+0000 '>' 2026-02-22T20:43:08.695032+0000 2026-02-22T20:43:14.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:43:15.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:43:15.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:15.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:43:15.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:15.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:43:15.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:15.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:43:15.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:43:08.695032+0000 '>' 2026-02-22T20:43:08.695032+0000 2026-02-22T20:43:15.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:43:16.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:43:16.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:16.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:43:16.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:16.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:43:16.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:16.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:43:16.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:43:08.695032+0000 '>' 2026-02-22T20:43:08.695032+0000 2026-02-22T20:43:16.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:43:17.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:43:17.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:17.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:43:17.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:17.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:43:17.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:17.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:43:17.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:43:14.793192+0000 '>' 2026-02-22T20:43:08.695032+0000 2026-02-22T20:43:17.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:43:17.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3711: TEST_corrupt_scrub_replicated: wait_for_clean 2026-02-22T20:43:17.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:43:17.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:43:17.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:43:17.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:43:17.894 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:43:17.894 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:43:17.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:43:17.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:43:17.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:43:18.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:43:18.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:43:18.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:43:18.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:43:18.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:43:18.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:43:18.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:43:18.164 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:43:18.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:43:18.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:18.164 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:43:18.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=850403524617 2026-02-22T20:43:18.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 850403524617 2026-02-22T20:43:18.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524617' 2026-02-22T20:43:18.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:18.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:43:18.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=871878361095 2026-02-22T20:43:18.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 871878361095 2026-02-22T20:43:18.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-850403524617 1-871878361095' 2026-02-22T20:43:18.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:18.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-850403524617 2026-02-22T20:43:18.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:18.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:43:18.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-850403524617 2026-02-22T20:43:18.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:18.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=850403524617 2026-02-22T20:43:18.316 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 850403524617 2026-02-22T20:43:18.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 850403524617' 2026-02-22T20:43:18.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:18.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524616 -lt 850403524617 2026-02-22T20:43:18.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:43:19.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:43:19.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:19.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524616 -lt 850403524617 2026-02-22T20:43:19.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:43:20.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:43:20.652 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:20.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 850403524617 -lt 850403524617 2026-02-22T20:43:20.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:20.810 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-871878361095 2026-02-22T20:43:20.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:20.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:43:20.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-871878361095 2026-02-22T20:43:20.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:20.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=871878361095 2026-02-22T20:43:20.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 871878361095' 2026-02-22T20:43:20.815 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 871878361095 2026-02-22T20:43:20.816 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:43:20.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 871878361095 -lt 871878361095 2026-02-22T20:43:20.976 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:43:20.976 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:43:20.976 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:43:21.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:43:21.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:43:21.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:43:21.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:43:21.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:43:21.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:43:21.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:43:21.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:43:21.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:43:21.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:43:21.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:43:21.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:43:21.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:43:21.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:43:21.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:43:21.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3714: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ17 td/osd-scrub-repair/robj17.out 2026-02-22T20:43:21.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3715: TEST_corrupt_scrub_replicated: timeout 30 rados -p csr_pool get ROBJ18 td/osd-scrub-repair/robj18.out 2026-02-22T20:43:21.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3717: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2026-02-22T20:43:21.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3718: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ17 td/osd-scrub-repair/robj17.out 2026-02-22T20:43:21.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3719: TEST_corrupt_scrub_replicated: diff -q td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2026-02-22T20:43:21.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3720: TEST_corrupt_scrub_replicated: rm -f td/osd-scrub-repair/new.ROBJ18 td/osd-scrub-repair/robj18.out 2026-02-22T20:43:21.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3722: TEST_corrupt_scrub_replicated: '[' 0 '!=' 0 ']' 2026-02-22T20:43:21.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:3728: TEST_corrupt_scrub_replicated: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2026-02-22T20:43:21.780 INFO:tasks.workunit.client.0.trial150.stderr:pool 'csr_pool' removed 2026-02-22T20:43:21.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:43:21.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:43:21.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:43:21.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:43:21.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:43:21.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:43:21.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:43:21.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:43:21.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:43:21.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:43:21.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:43:21.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:43:21.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:43:21.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:43:21.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:43:21.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:43:21.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:43:21.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:43:21.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:43:21.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:43:21.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:43:21.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:43:21.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:43:21.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:43:21.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:21.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:21.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:43:21.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:43:21.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:43:21.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:43:21.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:43:21.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:43:21.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:43:21.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:43:21.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:43:21.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:43:21.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:43:21.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:43:21.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:43:21.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:43:21.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:43:21.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:43:21.951 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:43:21.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:43:21.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:43:21.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:43:21.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:43:21.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:43:21.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:43:21.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:43:21.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:43:21.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:43:21.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:43:21.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:43:21.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:43:21.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:43:21.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:21.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:21.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:43:21.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:43:21.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:43:21.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:43:21.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:43:21.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:21.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:21.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:43:21.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:43:21.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:43:21.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:43:21.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:43:21.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_corrupt_snapset_scrub_rep td/osd-scrub-repair 2026-02-22T20:43:21.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6100: TEST_corrupt_snapset_scrub_rep: local dir=td/osd-scrub-repair 2026-02-22T20:43:21.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6101: TEST_corrupt_snapset_scrub_rep: local poolname=csr_pool 2026-02-22T20:43:21.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6102: TEST_corrupt_snapset_scrub_rep: local total_objs=2 2026-02-22T20:43:21.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6104: TEST_corrupt_snapset_scrub_rep: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-22T20:43:21.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:43:21.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:43:21.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:43:21.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:43:21.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:43:21.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-22T20:43:21.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:43:21.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:43:21.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:43:21.990 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:43:21.991 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:21.991 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:21.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:21.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-22T20:43:22.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:43:22.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:43:22.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:43:22.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:43:22.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:43:22.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:43:22.029 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:43:22.030 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:43:22.030 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:43:22.030 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:43:22.030 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:22.030 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:22.030 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:43:22.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:43:22.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:43:22.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:43:22.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:43:22.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:43:22.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:43:22.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:43:22.087 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:43:22.087 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:43:22.087 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:43:22.088 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:43:22.088 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:22.088 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:22.088 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:43:22.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:43:22.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:43:22.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6105: TEST_corrupt_snapset_scrub_rep: run_mgr td/osd-scrub-repair x 2026-02-22T20:43:22.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:43:22.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:43:22.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:43:22.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:43:22.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:43:22.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:43:22.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:43:22.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:43:22.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:43:22.238 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:43:22.238 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:22.238 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:22.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:22.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:43:22.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6106: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 0 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:43:22.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:43:22.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:43:22.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:43:22.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:43:22.260 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:43:22.260 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:22.260 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:22.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:43:22.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:43:22.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:43:22.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6346752f-1a89-418f-b6c1-fc33ee1b39d9 2026-02-22T20:43:22.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6346752f-1a89-418f-b6c1-fc33ee1b39d9' 2026-02-22T20:43:22.263 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 6346752f-1a89-418f-b6c1-fc33ee1b39d9 2026-02-22T20:43:22.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:43:22.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBqaptpnifHDxAAcVyT4balPcK4Kz8hgZA/Ug== 2026-02-22T20:43:22.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBqaptpnifHDxAAcVyT4balPcK4Kz8hgZA/Ug=="}' 2026-02-22T20:43:22.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6346752f-1a89-418f-b6c1-fc33ee1b39d9 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:43:22.365 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:43:22.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:43:22.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqaptpnifHDxAAcVyT4balPcK4Kz8hgZA/Ug== --osd-uuid 6346752f-1a89-418f-b6c1-fc33ee1b39d9 2026-02-22T20:43:22.392 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:22.377+0000 7f4561e1fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:22.393 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:22.381+0000 7f4561e1fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:22.394 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:22.381+0000 7f4561e1fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:22.395 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:22.381+0000 7f4561e1fa40 -1 bdev(0x55b5ba31cc00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:43:22.395 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:22.381+0000 7f4561e1fa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:43:23.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:43:23.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:43:23.128 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:43:23.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:43:23.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:43:23.335 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:43:23.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:43:23.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:43:23.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:43:23.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:43:23.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:43:23.352 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:23.337+0000 7f4ea5a3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:23.354 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:23.341+0000 7f4ea5a3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:23.355 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:23.341+0000 7f4ea5a3fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:23.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:43:23.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:43:23.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:43:23.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:43:23.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:43:23.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:43:23.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:43:23.496 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:43:23.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:43:23.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:43:23.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:43:23.774 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:23.761+0000 7f4ea5a3fa40 -1 Falling back to public interface 2026-02-22T20:43:24.314 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:24.301+0000 7f4ea5a3fa40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:43:24.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:43:24.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:43:24.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:43:24.656 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:43:24.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:43:24.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:43:24.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:43:25.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:43:25.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:43:25.815 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:43:25.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:43:25.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:43:25.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:43:25.971 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/18678863,v1:127.0.0.1:6803/18678863] [v2:127.0.0.1:6804/18678863,v1:127.0.0.1:6805/18678863] exists,up 6346752f-1a89-418f-b6c1-fc33ee1b39d9 2026-02-22T20:43:25.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:43:25.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:43:25.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:43:25.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6107: TEST_corrupt_snapset_scrub_rep: run_osd td/osd-scrub-repair 1 2026-02-22T20:43:25.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:43:25.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:43:25.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:43:25.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:43:25.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:43:25.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:43:25.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:43:25.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:43:25.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:43:25.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:43:25.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:43:25.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:43:25.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:43:25.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:43:25.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:43:25.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:43:25.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:43:25.982 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:43:25.982 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:25.982 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:25.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:25.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:25.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:43:25.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:43:25.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:43:25.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:43:25.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:43:25.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:43:25.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:43:25.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:43:25.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:43:25.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:43:25.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:43:25.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:43:25.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:43:25.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7d7e5034-32f6-4024-94fa-bd12d2c5867a 2026-02-22T20:43:25.985 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 7d7e5034-32f6-4024-94fa-bd12d2c5867a 2026-02-22T20:43:25.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7d7e5034-32f6-4024-94fa-bd12d2c5867a' 2026-02-22T20:43:25.986 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:43:25.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBtaptpE4ZaOhAAPFSNmczaFhXEp2yzhoFagg== 2026-02-22T20:43:25.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBtaptpE4ZaOhAAPFSNmczaFhXEp2yzhoFagg=="}' 2026-02-22T20:43:25.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7d7e5034-32f6-4024-94fa-bd12d2c5867a -i td/osd-scrub-repair/1/new.json 2026-02-22T20:43:26.141 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:43:26.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:43:26.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtaptpE4ZaOhAAPFSNmczaFhXEp2yzhoFagg== --osd-uuid 7d7e5034-32f6-4024-94fa-bd12d2c5867a 2026-02-22T20:43:26.170 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:26.157+0000 7f836d2d5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:26.171 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:26.157+0000 7f836d2d5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:26.172 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:26.157+0000 7f836d2d5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:26.173 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:26.161+0000 7f836d2d5a40 -1 bdev(0x557e2d764c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:43:26.173 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:26.161+0000 7f836d2d5a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:43:26.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:43:26.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:43:26.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:43:26.838 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:43:26.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:43:27.034 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:43:27.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:43:27.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:43:27.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:43:27.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:43:27.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:43:27.058 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:27.045+0000 7fc14a16ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:27.059 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:27.045+0000 7fc14a16ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:27.061 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:27.045+0000 7fc14a16ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:27.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:43:27.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:43:27.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:43:27.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:43:27.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:43:27.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:43:27.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:43:27.201 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:43:27.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:43:27.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:43:27.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:43:27.467 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:27.453+0000 7fc14a16ba40 -1 Falling back to public interface 2026-02-22T20:43:28.108 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:28.093+0000 7fc14a16ba40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:43:28.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:43:28.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:43:28.359 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:43:28.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:43:28.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:43:28.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:43:28.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:43:29.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:43:29.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:43:29.516 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:43:29.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:43:29.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:43:29.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:43:29.672 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/210014511,v1:127.0.0.1:6811/210014511] [v2:127.0.0.1:6812/210014511,v1:127.0.0.1:6813/210014511] exists,up 7d7e5034-32f6-4024-94fa-bd12d2c5867a 2026-02-22T20:43:29.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:43:29.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:43:29.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:43:29.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6108: TEST_corrupt_snapset_scrub_rep: create_rbd_pool 2026-02-22T20:43:29.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:43:29.822 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:43:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:43:29.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:43:30.033 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:43:30.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:43:31.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:43:31.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6109: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2026-02-22T20:43:31.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:43:31.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:43:31.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:43:31.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:43:31.326 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:43:31.326 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:43:31.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:43:31.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:43:31.327 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:43:31.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:43:31.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:43:31.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:43:31.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:43:31.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:43:31.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:43:31.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:43:31.601 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:43:31.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:43:31.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:31.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:43:31.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:43:31.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:43:31.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:43:31.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:31.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:43:31.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-22T20:43:31.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-22T20:43:31.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-22T20:43:31.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:31.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:43:31.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:31.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:43:31.749 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:43:31.749 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:31.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:43:31.751 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:43:31.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:43:31.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:31.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:43:31.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:43:32.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:43:32.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:33.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T20:43:33.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:33.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-22T20:43:33.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:33.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:43:33.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-22T20:43:33.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:33.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-22T20:43:33.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-22T20:43:33.089 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705666 2026-02-22T20:43:33.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:43:33.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-22T20:43:33.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:43:33.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:43:33.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:43:33.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:43:33.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:43:33.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:43:33.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:43:33.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:43:33.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:43:33.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:43:33.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:43:33.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:43:33.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:43:33.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:43:33.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:43:33.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:43:33.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:43:33.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:43:33.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6111: TEST_corrupt_snapset_scrub_rep: create_pool foo 1 2026-02-22T20:43:33.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-22T20:43:34.003 INFO:tasks.workunit.client.0.trial150.stderr:pool 'foo' created 2026-02-22T20:43:34.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:43:35.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6112: TEST_corrupt_snapset_scrub_rep: create_pool csr_pool 1 1 2026-02-22T20:43:35.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2026-02-22T20:43:35.231 INFO:tasks.workunit.client.0.trial150.stderr:pool 'csr_pool' created 2026-02-22T20:43:35.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:43:36.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6113: TEST_corrupt_snapset_scrub_rep: wait_for_clean 2026-02-22T20:43:36.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:43:36.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:43:36.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:43:36.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:43:36.245 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:43:36.245 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:43:36.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:43:36.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:43:36.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:43:36.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:43:36.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:43:36.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:43:36.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:43:36.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:43:36.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:43:36.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:43:36.524 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:43:36.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:43:36.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:36.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:43:36.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-22T20:43:36.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-22T20:43:36.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-22T20:43:36.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:36.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:43:36.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705668 2026-02-22T20:43:36.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705668 2026-02-22T20:43:36.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705668' 2026-02-22T20:43:36.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:36.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-22T20:43:36.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:36.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:43:36.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-22T20:43:36.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:36.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-22T20:43:36.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-22T20:43:36.677 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836485 2026-02-22T20:43:36.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:36.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-22T20:43:36.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:36.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705668 2026-02-22T20:43:36.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:36.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:43:36.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705668 2026-02-22T20:43:36.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:36.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705668 2026-02-22T20:43:36.843 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705668 2026-02-22T20:43:36.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-22T20:43:36.844 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:43:37.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705668 2026-02-22T20:43:37.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:43:37.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:43:37.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:43:37.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:43:37.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:43:37.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:43:37.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:43:37.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:43:37.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:43:37.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:43:37.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:43:37.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:43:37.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:43:37.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:43:37.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:43:37.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:43:37.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:43:37.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:43:37.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6115: TEST_corrupt_snapset_scrub_rep: seq 1 2 2026-02-22T20:43:37.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6115: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2026-02-22T20:43:37.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6116: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2026-02-22T20:43:37.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6117: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ1 2026-02-22T20:43:37.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:43:37.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:43:37.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2026-02-22T20:43:37.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:43:37.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:43:37.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:43:37.754 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:43:37.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:43:37.958 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:43:37.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:43:37.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:43:37.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:43:37.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6119: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2026-02-22T20:43:38.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6120: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2026-02-22T20:43:38.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6115: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2026-02-22T20:43:38.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6116: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2026-02-22T20:43:38.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6117: TEST_corrupt_snapset_scrub_rep: add_something td/osd-scrub-repair csr_pool ROBJ2 2026-02-22T20:43:38.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:43:38.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:43:38.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2026-02-22T20:43:38.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:43:38.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:43:38.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:43:38.249 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:43:38.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:43:38.453 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:43:38.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:43:38.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:43:38.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:43:38.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6119: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2026-02-22T20:43:38.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6120: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2026-02-22T20:43:38.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6123: TEST_corrupt_snapset_scrub_rep: get_pg csr_pool ROBJ0 2026-02-22T20:43:38.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-22T20:43:38.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-22T20:43:38.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-22T20:43:38.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:43:38.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6123: TEST_corrupt_snapset_scrub_rep: local pg=3.0 2026-02-22T20:43:38.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6124: TEST_corrupt_snapset_scrub_rep: get_primary csr_pool ROBJ0 2026-02-22T20:43:38.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2026-02-22T20:43:38.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2026-02-22T20:43:38.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2026-02-22T20:43:38.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:43:38.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6124: TEST_corrupt_snapset_scrub_rep: local primary=1 2026-02-22T20:43:38.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6126: TEST_corrupt_snapset_scrub_rep: rados -p csr_pool mksnap snap1 2026-02-22T20:43:38.885 INFO:tasks.workunit.client.0.trial150.stdout:created pool csr_pool snap snap1 2026-02-22T20:43:38.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6127: TEST_corrupt_snapset_scrub_rep: echo -n head_of_snapshot_data 2026-02-22T20:43:38.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6129: TEST_corrupt_snapset_scrub_rep: seq 1 2 2026-02-22T20:43:38.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6129: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2026-02-22T20:43:38.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6130: TEST_corrupt_snapset_scrub_rep: objname=ROBJ1 2026-02-22T20:43:38.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6133: TEST_corrupt_snapset_scrub_rep: expr 1 % 2 2026-02-22T20:43:38.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6133: TEST_corrupt_snapset_scrub_rep: local osd=1 2026-02-22T20:43:38.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6135: TEST_corrupt_snapset_scrub_rep: case $i in 2026-02-22T20:43:38.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6137: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/change 2026-02-22T20:43:38.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6138: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2026-02-22T20:43:38.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:43:38.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:43:38.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:43:38.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:43:38.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2026-02-22T20:43:38.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:43:38.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:43:38.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:43:38.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:43:38.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:43:38.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:43:38.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:43:38.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:43:38.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:43:38.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:43:39.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:43:39.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 --head ROBJ1 clear-snapset corrupt 2026-02-22T20:43:39.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:43:39.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:43:39.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:43:39.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:43:39.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:43:39.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 --head ROBJ1 clear-snapset corrupt 2026-02-22T20:43:39.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 2026-02-22T20:43:39.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:39.498 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:39.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:39.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:39.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:43:39.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:43:39.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:43:39.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:43:39.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:43:39.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:43:39.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:43:39.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:43:39.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:43:39.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:43:39.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:43:39.500 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:43:39.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:43:39.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:43:39.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:43:39.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:43:39.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:43:39.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:43:39.515 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:39.501+0000 7fd5d03f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:39.517 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:39.505+0000 7fd5d03f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:39.518 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:39.505+0000 7fd5d03f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:39.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:43:39.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:43:39.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:43:39.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:43:39.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:43:39.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:43:39.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:43:39.667 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:43:39.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:43:39.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:43:39.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:43:39.887 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:39.873+0000 7fd5d03f9a40 -1 Falling back to public interface 2026-02-22T20:43:40.465 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:40.453+0000 7fd5d03f9a40 -1 osd.1 29 log_to_monitors true 2026-02-22T20:43:40.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:43:40.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:43:40.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:43:40.830 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:43:40.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:43:40.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:43:40.983 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 33 up_thru 33 down_at 30 last_clean_interval [9,29) [v2:127.0.0.1:6810/2925551799,v1:127.0.0.1:6811/2925551799] [v2:127.0.0.1:6812/2925551799,v1:127.0.0.1:6813/2925551799] exists,up 7d7e5034-32f6-4024-94fa-bd12d2c5867a 2026-02-22T20:43:40.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:43:40.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:43:40.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:43:40.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:43:40.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:43:40.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:43:40.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:43:40.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:43:40.984 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:43:40.984 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:43:40.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:43:40.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:43:40.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:43:41.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:43:41.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:43:41.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:43:41.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:43:41.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:43:41.107 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:43:41.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:43:41.272 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:43:41.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:43:41.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:41.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:43:41.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-22T20:43:41.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-22T20:43:41.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-22T20:43:41.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:41.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:43:41.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920770 2026-02-22T20:43:41.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920770 2026-02-22T20:43:41.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-141733920770' 2026-02-22T20:43:41.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:41.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-22T20:43:41.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:41.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:43:41.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-22T20:43:41.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:41.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-22T20:43:41.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-22T20:43:41.422 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836487 2026-02-22T20:43:41.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:41.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T20:43:41.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:43:42.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:43:42.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:42.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T20:43:42.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:43:43.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:43:43.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:43.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-22T20:43:43.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:43.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-141733920770 2026-02-22T20:43:43.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:43.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:43:43.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-141733920770 2026-02-22T20:43:43.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:43.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=141733920770 2026-02-22T20:43:43.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 141733920770' 2026-02-22T20:43:43.905 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 141733920770 2026-02-22T20:43:43.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:43:44.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 141733920770 -lt 141733920770 2026-02-22T20:43:44.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:43:44.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:43:44.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:43:44.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:43:44.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:43:44.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:43:44.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:43:44.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:43:44.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:43:44.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:43:44.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:43:44.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:43:44.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:43:44.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:43:44.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:43:44.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:43:44.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:43:44.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:43:44.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6129: TEST_corrupt_snapset_scrub_rep: for i in $(seq 1 $total_objs) 2026-02-22T20:43:44.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6130: TEST_corrupt_snapset_scrub_rep: objname=ROBJ2 2026-02-22T20:43:44.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6133: TEST_corrupt_snapset_scrub_rep: expr 2 % 2 2026-02-22T20:43:44.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6133: TEST_corrupt_snapset_scrub_rep: local osd=0 2026-02-22T20:43:44.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6135: TEST_corrupt_snapset_scrub_rep: case $i in 2026-02-22T20:43:44.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6142: TEST_corrupt_snapset_scrub_rep: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/change 2026-02-22T20:43:44.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6143: TEST_corrupt_snapset_scrub_rep: objectstore_tool td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:43:44.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:43:44.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:43:44.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 --head ROBJ2 clear-snapset corrupt 2026-02-22T20:43:44.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:43:44.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:43:44.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:43:44.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:43:44.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:43:44.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 --head ROBJ2 clear-snapset corrupt 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:43:45.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:43:45.271 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:43:45.271 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:45.271 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:45.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:43:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:43:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:43:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:43:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:43:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:43:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:43:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:43:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:43:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:43:45.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:43:45.273 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:43:45.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:43:45.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:43:45.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:43:45.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:43:45.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:43:45.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:43:45.292 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:45.277+0000 7f0ceeaaba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:45.294 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:45.281+0000 7f0ceeaaba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:45.295 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:45.281+0000 7f0ceeaaba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:45.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:43:45.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:43:45.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:43:45.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:43:45.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:43:45.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:43:45.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:43:45.436 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:43:45.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:43:45.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:43:45.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:43:45.775 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:45.761+0000 7f0ceeaaba40 -1 Falling back to public interface 2026-02-22T20:43:46.411 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:46.397+0000 7f0ceeaaba40 -1 osd.0 34 log_to_monitors true 2026-02-22T20:43:46.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:43:46.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:43:46.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:43:46.599 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:43:46.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:43:46.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:43:46.759 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 35 last_clean_interval [5,34) [v2:127.0.0.1:6802/1863905987,v1:127.0.0.1:6803/1863905987] [v2:127.0.0.1:6804/1863905987,v1:127.0.0.1:6805/1863905987] exists,up 6346752f-1a89-418f-b6c1-fc33ee1b39d9 2026-02-22T20:43:46.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:43:46.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:43:46.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:43:46.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:43:46.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:43:46.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:43:46.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:43:46.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:43:46.761 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:43:46.761 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:43:46.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:43:46.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:43:46.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:43:46.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:43:46.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:43:46.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:43:46.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:43:46.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:43:46.882 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:43:47.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:43:47.042 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:43:47.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:43:47.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:47.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:43:47.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757250 2026-02-22T20:43:47.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757250 2026-02-22T20:43:47.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757250' 2026-02-22T20:43:47.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:47.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:43:47.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920772 2026-02-22T20:43:47.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920772 2026-02-22T20:43:47.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757250 1-141733920772' 2026-02-22T20:43:47.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:47.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-163208757250 2026-02-22T20:43:47.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:47.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:43:47.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-163208757250 2026-02-22T20:43:47.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:47.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757250 2026-02-22T20:43:47.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 163208757250' 2026-02-22T20:43:47.197 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 163208757250 2026-02-22T20:43:47.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:47.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 163208757250 2026-02-22T20:43:47.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:43:48.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:43:48.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:48.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 163208757250 2026-02-22T20:43:48.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:43:49.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:43:49.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:49.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757250 -lt 163208757250 2026-02-22T20:43:49.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:49.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-141733920772 2026-02-22T20:43:49.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:49.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:43:49.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-141733920772 2026-02-22T20:43:49.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:49.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=141733920772 2026-02-22T20:43:49.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 141733920772' 2026-02-22T20:43:49.693 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 141733920772 2026-02-22T20:43:49.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:43:49.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 141733920772 -lt 141733920772 2026-02-22T20:43:49.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:43:49.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:43:49.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:43:50.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:43:50.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:43:50.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:43:50.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:43:50.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:43:50.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:43:50.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:43:50.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:43:50.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:43:50.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:43:50.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:43:50.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:43:50.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:43:50.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:43:50.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:43:50.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6148: TEST_corrupt_snapset_scrub_rep: rm td/osd-scrub-repair/change 2026-02-22T20:43:50.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6150: TEST_corrupt_snapset_scrub_rep: pg_scrub 3.0 2026-02-22T20:43:50.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-22T20:43:50.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-22T20:43:50.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:43:50.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:43:50.408 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:43:50.408 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:43:50.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:43:50.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:43:50.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:43:50.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:43:50.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:43:50.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:43:50.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:43:50.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:43:50.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:43:50.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:43:50.880 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:43:50.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:43:50.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:50.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:43:50.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757251 2026-02-22T20:43:50.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757251 2026-02-22T20:43:50.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757251' 2026-02-22T20:43:50.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:43:50.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:43:51.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920774 2026-02-22T20:43:51.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920774 2026-02-22T20:43:51.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757251 1-141733920774' 2026-02-22T20:43:51.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:51.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-163208757251 2026-02-22T20:43:51.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:51.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:43:51.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-163208757251 2026-02-22T20:43:51.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:51.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757251 2026-02-22T20:43:51.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 163208757251' 2026-02-22T20:43:51.030 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 163208757251 2026-02-22T20:43:51.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:51.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757250 -lt 163208757251 2026-02-22T20:43:51.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:43:52.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:43:52.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:52.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757250 -lt 163208757251 2026-02-22T20:43:52.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:43:53.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:43:53.353 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:43:53.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757252 -lt 163208757251 2026-02-22T20:43:53.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:43:53.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-141733920774 2026-02-22T20:43:53.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:43:53.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:43:53.518 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-141733920774 2026-02-22T20:43:53.518 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:43:53.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=141733920774 2026-02-22T20:43:53.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 141733920774' 2026-02-22T20:43:53.520 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 141733920774 2026-02-22T20:43:53.520 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:43:53.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 141733920774 -lt 141733920774 2026-02-22T20:43:53.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:43:53.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:43:53.690 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:43:53.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:43:53.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:43:53.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:43:53.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:43:53.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:43:53.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T20:43:53.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:43:53.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:43:53.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:43:53.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-22T20:43:53.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:53.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:43:53.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:53.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:43:53.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-22T20:43:35.220958+0000 2026-02-22T20:43:53.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-22T20:43:54.076 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-22T20:43:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-22T20:43:35.220958+0000 2026-02-22T20:43:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:43:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:43:35.220958+0000 2026-02-22T20:43:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:43:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:43:54.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:54.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:43:54.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:54.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:43:54.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:54.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:43:54.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:43:35.220958+0000 '>' 2026-02-22T20:43:35.220958+0000 2026-02-22T20:43:54.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:43:55.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:43:55.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:55.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:43:55.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:55.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:43:55.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:55.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:43:55.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:43:35.220958+0000 '>' 2026-02-22T20:43:35.220958+0000 2026-02-22T20:43:55.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:43:56.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:43:56.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:56.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:43:56.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:56.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:43:56.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:56.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:43:56.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:43:35.220958+0000 '>' 2026-02-22T20:43:35.220958+0000 2026-02-22T20:43:56.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:43:57.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:43:57.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:43:57.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:43:57.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:43:57.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:43:57.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:43:57.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:43:57.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:43:54.351798+0000 '>' 2026-02-22T20:43:35.220958+0000 2026-02-22T20:43:57.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:43:57.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6152: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-pg csr_pool 2026-02-22T20:43:57.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6154: TEST_corrupt_snapset_scrub_rep: jq '. | length' td/osd-scrub-repair/json 2026-02-22T20:43:57.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6154: TEST_corrupt_snapset_scrub_rep: test 1 = 1 2026-02-22T20:43:57.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6156: TEST_corrupt_snapset_scrub_rep: jq -r '.[0]' td/osd-scrub-repair/json 2026-02-22T20:43:57.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6156: TEST_corrupt_snapset_scrub_rep: test 3.0 = 3.0 2026-02-22T20:43:57.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6158: TEST_corrupt_snapset_scrub_rep: rados list-inconsistent-obj 3.0 2026-02-22T20:43:57.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2026-02-22T20:43:57.778 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:43:57.778 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:43:57.778 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:43:57.778 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:43:57.778 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:43:57.779 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:43:57.779 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:43:57.779 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:43:57.779 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:43:57.779 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:43:57.779 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:43:57.779 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:43:57.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2026-02-22T20:43:57.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6160: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:43:57.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6329: TEST_corrupt_snapset_scrub_rep: jq 'def walk(f): 2026-02-22T20:43:57.797 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:43:57.797 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:43:57.797 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:43:57.797 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:43:57.797 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:43:57.797 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:43:57.797 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:43:57.798 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:43:57.798 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:43:57.798 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:43:57.798 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:43:57.798 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' td/osd-scrub-repair/json 2026-02-22T20:43:57.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6329: TEST_corrupt_snapset_scrub_rep: jq .inconsistents 2026-02-22T20:43:57.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6329: TEST_corrupt_snapset_scrub_rep: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:43:57.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6330: TEST_corrupt_snapset_scrub_rep: multidiff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:43:57.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-repair/checkcsjson td/osd-scrub-repair/csjson 2026-02-22T20:43:57.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6331: TEST_corrupt_snapset_scrub_rep: test no = yes 2026-02-22T20:43:57.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6336: TEST_corrupt_snapset_scrub_rep: test '' = yes 2026-02-22T20:43:57.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6341: TEST_corrupt_snapset_scrub_rep: ERRORS=0 2026-02-22T20:43:57.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6342: TEST_corrupt_snapset_scrub_rep: declare -a err_strings 2026-02-22T20:43:57.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6343: TEST_corrupt_snapset_scrub_rep: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' 2026-02-22T20:43:57.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6344: TEST_corrupt_snapset_scrub_rep: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' 2026-02-22T20:43:57.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6345: TEST_corrupt_snapset_scrub_rep: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' 2026-02-22T20:43:57.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6346: TEST_corrupt_snapset_scrub_rep: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' 2026-02-22T20:43:57.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6347: TEST_corrupt_snapset_scrub_rep: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' 2026-02-22T20:43:57.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6348: TEST_corrupt_snapset_scrub_rep: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' 2026-02-22T20:43:57.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-22T20:43:57.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ1:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:57.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-22T20:43:57.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid [0-9]*:.*:::ROBJ2:head : snapset inconsistent' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:57.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-22T20:43:57.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 [0-9]*:.*:::ROBJ1:1 : is an unexpected clone' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:57.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-22T20:43:57.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub : stat mismatch, got 3/4 objects, 1/2 clones, 3/4 dirty, 3/4 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 49/56 bytes, 0/0 manifest objects, 0/0 hit_set_archive bytes.' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:57.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-22T20:43:57.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 0 missing, 2 inconsistent objects' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:57.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6350: TEST_corrupt_snapset_scrub_rep: for err_string in "${err_strings[@]}" 2026-02-22T20:43:57.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6352: TEST_corrupt_snapset_scrub_rep: grep -q 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 4 errors' td/osd-scrub-repair/osd.1.log 2026-02-22T20:43:57.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6359: TEST_corrupt_snapset_scrub_rep: '[' 0 '!=' 0 ']' 2026-02-22T20:43:57.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6365: TEST_corrupt_snapset_scrub_rep: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2026-02-22T20:43:58.042 INFO:tasks.workunit.client.0.trial150.stderr:pool 'csr_pool' removed 2026-02-22T20:43:58.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:43:58.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:43:58.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:43:58.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:43:58.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:43:58.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:43:58.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:43:58.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:43:58.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:43:58.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:43:58.164 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:43:58.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:43:58.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:43:58.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:43:58.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:43:58.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:43:58.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:43:58.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:43:58.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:43:58.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:43:58.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:43:58.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:43:58.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:43:58.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:43:58.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:58.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:58.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:43:58.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:43:58.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:43:58.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:43:58.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:43:58.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:43:58.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:43:58.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:43:58.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:43:58.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:43:58.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:43:58.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:43:58.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:43:58.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:43:58.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:43:58.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:43:58.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:43:58.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:43:58.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:43:58.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:43:58.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:43:58.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:43:58.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:43:58.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:43:58.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:43:58.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:43:58.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:43:58.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:43:58.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:43:58.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:43:58.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:58.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:58.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:43:58.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:43:58.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:43:58.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:43:58.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:43:58.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:58.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:58.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:43:58.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_dual_store_replicated_cluster td/osd-scrub-repair 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6432: TEST_dual_store_replicated_cluster: local dir=td/osd-scrub-repair 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6433: TEST_dual_store_replicated_cluster: local poolname=csr_pool 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6434: TEST_dual_store_replicated_cluster: local total_objs=19 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6435: TEST_dual_store_replicated_cluster: local extr_dbg=1 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6437: TEST_dual_store_replicated_cluster: run_mon td/osd-scrub-repair a --osd_pool_default_size=2 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:43:58.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=2 2026-02-22T20:43:58.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:43:58.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:43:58.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:43:58.227 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:43:58.228 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:58.228 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:58.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:58.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-22T20:43:58.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:43:58.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:43:58.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:43:58.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:43:58.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:43:58.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:43:58.259 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:43:58.259 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:43:58.259 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:43:58.259 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:43:58.259 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:58.259 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:58.259 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:43:58.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:43:58.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:43:58.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:43:58.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:43:58.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:43:58.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:43:58.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:43:58.316 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:43:58.316 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:43:58.316 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:43:58.317 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:43:58.317 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:58.317 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:58.317 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:43:58.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:43:58.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:43:58.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6438: TEST_dual_store_replicated_cluster: run_mgr td/osd-scrub-repair x --mgr_stats_period=1 2026-02-22T20:43:58.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:43:58.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:43:58.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:43:58.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:43:58.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:43:58.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:43:58.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:43:58.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:43:58.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:43:58.464 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:43:58.464 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:58.464 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:58.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:58.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:43:58.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-22T20:43:58.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6439: TEST_dual_store_replicated_cluster: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2026-02-22T20:43:58.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6440: TEST_dual_store_replicated_cluster: ceph_osd_args+='--osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 ' 2026-02-22T20:43:58.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6441: TEST_dual_store_replicated_cluster: ceph_osd_args+='--osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 ' 2026-02-22T20:43:58.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6442: TEST_dual_store_replicated_cluster: seq 0 1 2026-02-22T20:43:58.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6442: TEST_dual_store_replicated_cluster: for osd in $(seq 0 1) 2026-02-22T20:43:58.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6444: TEST_dual_store_replicated_cluster: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:43:58.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:43:58.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:43:58.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:43:58.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:43:58.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:43:58.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:43:58.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:43:58.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:58.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:43:58.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:43:58.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:43:58.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:43:58.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:43:58.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:43:58.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:43:58.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:43:58.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:43:58.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:43:58.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:43:58.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:43:58.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:43:58.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:43:58.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=976f5525-b691-48b1-b2d2-8b61723446d8 2026-02-22T20:43:58.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 976f5525-b691-48b1-b2d2-8b61723446d8' 2026-02-22T20:43:58.491 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 976f5525-b691-48b1-b2d2-8b61723446d8 2026-02-22T20:43:58.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:43:58.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCOaptpl8ZkHRAAbPQPkr1UNyZYIFqltJZUjw== 2026-02-22T20:43:58.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCOaptpl8ZkHRAAbPQPkr1UNyZYIFqltJZUjw=="}' 2026-02-22T20:43:58.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 976f5525-b691-48b1-b2d2-8b61723446d8 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:43:58.594 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:43:58.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:43:58.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 --mkfs --key AQCOaptpl8ZkHRAAbPQPkr1UNyZYIFqltJZUjw== --osd-uuid 976f5525-b691-48b1-b2d2-8b61723446d8 2026-02-22T20:43:58.622 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:58.609+0000 7f401edd9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:58.624 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:58.609+0000 7f401edd9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:58.625 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:58.609+0000 7f401edd9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:58.625 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:58.613+0000 7f401edd9a40 -1 bdev(0x555b90db4c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:43:58.626 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:58.613+0000 7f401edd9a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:43:59.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:43:59.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:43:59.378 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:43:59.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:43:59.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:43:59.579 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:43:59.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:43:59.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:43:59.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:43:59.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:43:59.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:43:59.598 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:59.585+0000 7f090cdcfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:59.599 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:59.585+0000 7f090cdcfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:59.600 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:59.585+0000 7f090cdcfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:43:59.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:43:59.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:43:59.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:43:59.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:43:59.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:43:59.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:43:59.744 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:43:59.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:43:59.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:43:59.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:43:59.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:44:00.011 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:43:59.997+0000 7f090cdcfa40 -1 Falling back to public interface 2026-02-22T20:44:00.596 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:00.581+0000 7f090cdcfa40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:44:00.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:44:00.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:00.906 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:44:00.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:44:00.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:00.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:44:01.067 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2841613460,v1:127.0.0.1:6803/2841613460] [v2:127.0.0.1:6804/2841613460,v1:127.0.0.1:6805/2841613460] exists,up 976f5525-b691-48b1-b2d2-8b61723446d8 2026-02-22T20:44:01.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:44:01.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:44:01.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:44:01.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6442: TEST_dual_store_replicated_cluster: for osd in $(seq 0 1) 2026-02-22T20:44:01.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6444: TEST_dual_store_replicated_cluster: run_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:01.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:44:01.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:44:01.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:44:01.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:44:01.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:44:01.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:44:01.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:44:01.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:44:01.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:44:01.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:44:01.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:44:01.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:44:01.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:44:01.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:44:01.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:44:01.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:44:01.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:44:01.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:44:01.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:44:01.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:44:01.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:44:01.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:44:01.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:44:01.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:44:01.075 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:44:01.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6f50b59b-4813-480d-aedb-da74f6769de9' 2026-02-22T20:44:01.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:44:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCRaptpFdR0BBAAYhyzs8F/FCT898BFCFWBRQ== 2026-02-22T20:44:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCRaptpFdR0BBAAYhyzs8F/FCT898BFCFWBRQ=="}' 2026-02-22T20:44:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6f50b59b-4813-480d-aedb-da74f6769de9 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:44:01.245 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:44:01.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:44:01.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 --mkfs --key AQCRaptpFdR0BBAAYhyzs8F/FCT898BFCFWBRQ== --osd-uuid 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:44:01.282 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:01.269+0000 7f07ae7efa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:01.284 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:01.269+0000 7f07ae7efa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:01.285 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:01.269+0000 7f07ae7efa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:01.285 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:01.273+0000 7f07ae7efa40 -1 bdev(0x56068c18cc00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:44:01.286 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:01.273+0000 7f07ae7efa40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:44:01.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:44:01.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:44:01.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:44:01.934 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:44:01.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:44:02.131 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:44:02.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:44:02.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:02.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:44:02.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:44:02.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:44:02.159 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:02.145+0000 7fe17b798a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:02.161 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:02.145+0000 7fe17b798a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:02.162 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:02.149+0000 7fe17b798a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:02.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:44:02.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:44:02.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:44:02.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:44:02.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:44:02.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:02.291 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:44:02.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:44:02.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:02.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:44:02.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:44:02.515 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:02.501+0000 7fe17b798a40 -1 Falling back to public interface 2026-02-22T20:44:03.110 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:03.097+0000 7fe17b798a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:44:03.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:44:03.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:03.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:44:03.453 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:44:03.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:03.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:44:03.615 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4156864081,v1:127.0.0.1:6811/4156864081] [v2:127.0.0.1:6812/4156864081,v1:127.0.0.1:6813/4156864081] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:44:03.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:44:03.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:44:03.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:44:03.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6447: TEST_dual_store_replicated_cluster: create_rbd_pool 2026-02-22T20:44:03.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:44:03.767 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:44:03.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:44:03.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:44:03.995 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:44:04.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:44:05.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:44:05.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6448: TEST_dual_store_replicated_cluster: wait_for_clean 2026-02-22T20:44:05.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:44:05.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:44:05.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:44:05.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:44:05.279 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:44:05.279 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:44:05.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:44:05.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:44:05.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:44:05.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:44:05.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:44:05.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:44:05.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:44:05.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:44:05.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:44:05.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:44:05.546 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:44:05.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:44:05.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:05.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:44:05.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-22T20:44:05.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-22T20:44:05.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-22T20:44:05.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:05.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:44:05.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-22T20:44:05.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-22T20:44:05.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705667' 2026-02-22T20:44:05.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:05.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-22T20:44:05.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:05.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:44:05.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-22T20:44:05.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:05.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-22T20:44:05.700 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836485 2026-02-22T20:44:05.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-22T20:44:05.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:05.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-22T20:44:05.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:06.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:44:06.866 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:07.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836485 2026-02-22T20:44:07.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:07.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-22T20:44:07.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:07.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:44:07.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-22T20:44:07.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:07.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-22T20:44:07.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-22T20:44:07.033 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705667 2026-02-22T20:44:07.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:44:07.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705668 -lt 38654705667 2026-02-22T20:44:07.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:44:07.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:07.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:07.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:44:07.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:44:07.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:44:07.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:44:07.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:44:07.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:44:07.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:44:07.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:44:07.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:44:07.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:44:07.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:07.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:07.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:44:07.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:44:07.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:44:07.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6450: TEST_dual_store_replicated_cluster: create_pool foo 1 2026-02-22T20:44:07.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 1 2026-02-22T20:44:07.936 INFO:tasks.workunit.client.0.trial150.stderr:pool 'foo' created 2026-02-22T20:44:07.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:44:08.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6451: TEST_dual_store_replicated_cluster: create_pool csr_pool 1 1 2026-02-22T20:44:08.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create csr_pool 1 1 2026-02-22T20:44:09.117 INFO:tasks.workunit.client.0.trial150.stderr:pool 'csr_pool' created 2026-02-22T20:44:09.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:44:10.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6452: TEST_dual_store_replicated_cluster: wait_for_clean 2026-02-22T20:44:10.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:44:10.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:44:10.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:44:10.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:44:10.131 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:44:10.132 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:44:10.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:44:10.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:44:10.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:44:10.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:44:10.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:44:10.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:44:10.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:44:10.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:44:10.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:44:10.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:44:10.409 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:44:10.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:44:10.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:10.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:44:10.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-22T20:44:10.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-22T20:44:10.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T20:44:10.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:10.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:44:10.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705673 2026-02-22T20:44:10.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705673 2026-02-22T20:44:10.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705673' 2026-02-22T20:44:10.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:10.556 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-22T20:44:10.556 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:10.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:44:10.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-22T20:44:10.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:10.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-22T20:44:10.561 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836491 2026-02-22T20:44:10.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T20:44:10.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:10.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-22T20:44:10.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:11.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:44:11.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:11.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836491 2026-02-22T20:44:11.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:11.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705673 2026-02-22T20:44:11.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:11.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:44:11.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705673 2026-02-22T20:44:11.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:11.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705673 2026-02-22T20:44:11.896 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705673 2026-02-22T20:44:11.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705673' 2026-02-22T20:44:11.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:44:12.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705673 -lt 38654705673 2026-02-22T20:44:12.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:44:12.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:12.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:12.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:44:12.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:44:12.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:44:12.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:44:12.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:44:12.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:44:12.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:44:12.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:44:12.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:44:12.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:44:12.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:12.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:12.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:44:12.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:44:12.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:44:12.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6454: TEST_dual_store_replicated_cluster: ceph osd pool set csr_pool noscrub 1 2026-02-22T20:44:12.804 INFO:tasks.workunit.client.0.trial150.stderr:set pool 3 noscrub to 1 2026-02-22T20:44:12.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6455: TEST_dual_store_replicated_cluster: ceph osd pool set csr_pool nodeep-scrub 1 2026-02-22T20:44:13.009 INFO:tasks.workunit.client.0.trial150.stderr:set pool 3 nodeep-scrub to 1 2026-02-22T20:44:13.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: seq 1 19 2026-02-22T20:44:13.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:13.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ1 2026-02-22T20:44:13.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ1 2026-02-22T20:44:13.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:13.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:13.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ1 2026-02-22T20:44:13.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:13.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:13.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:13.212 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:13.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:13.412 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:13.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:13.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:13.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:13.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ1 hdr-ROBJ1 2026-02-22T20:44:13.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ1 key-ROBJ1 val-ROBJ1 2026-02-22T20:44:13.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:13.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ2 2026-02-22T20:44:13.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ2 2026-02-22T20:44:13.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:13.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:13.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ2 2026-02-22T20:44:13.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:13.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:13.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:13.699 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:13.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:13.904 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:13.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:13.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:13.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ2 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:13.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ2 hdr-ROBJ2 2026-02-22T20:44:13.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ2 key-ROBJ2 val-ROBJ2 2026-02-22T20:44:13.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:13.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ3 2026-02-22T20:44:13.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ3 2026-02-22T20:44:13.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:13.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:13.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ3 2026-02-22T20:44:13.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:13.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:13.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:14.181 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:14.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:14.385 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:14.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:14.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:14.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ3 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:14.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ3 hdr-ROBJ3 2026-02-22T20:44:14.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ3 key-ROBJ3 val-ROBJ3 2026-02-22T20:44:14.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:14.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ4 2026-02-22T20:44:14.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ4 2026-02-22T20:44:14.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:14.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:14.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ4 2026-02-22T20:44:14.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:14.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:14.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:14.671 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:14.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:14.877 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:14.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:14.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:14.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ4 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:14.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ4 hdr-ROBJ4 2026-02-22T20:44:14.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ4 key-ROBJ4 val-ROBJ4 2026-02-22T20:44:14.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:14.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ5 2026-02-22T20:44:14.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ5 2026-02-22T20:44:14.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:14.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:14.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ5 2026-02-22T20:44:14.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:14.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:14.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:15.166 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:15.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:15.371 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:15.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:15.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:15.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ5 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:15.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ5 hdr-ROBJ5 2026-02-22T20:44:15.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ5 key-ROBJ5 val-ROBJ5 2026-02-22T20:44:15.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:15.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ6 2026-02-22T20:44:15.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ6 2026-02-22T20:44:15.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:15.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:15.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ6 2026-02-22T20:44:15.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:15.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:15.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:15.674 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:15.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:15.877 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:15.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:15.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:15.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ6 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:15.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ6 hdr-ROBJ6 2026-02-22T20:44:15.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ6 key-ROBJ6 val-ROBJ6 2026-02-22T20:44:15.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:15.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ7 2026-02-22T20:44:15.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ7 2026-02-22T20:44:15.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:15.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:15.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ7 2026-02-22T20:44:15.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:15.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:15.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:16.176 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:16.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:16.381 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:16.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:16.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:16.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ7 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:16.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ7 hdr-ROBJ7 2026-02-22T20:44:16.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ7 key-ROBJ7 val-ROBJ7 2026-02-22T20:44:16.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:16.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ8 2026-02-22T20:44:16.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ8 2026-02-22T20:44:16.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:16.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:16.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ8 2026-02-22T20:44:16.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:16.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:16.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:16.676 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:16.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:16.881 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:16.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:16.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:16.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ8 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:16.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ8 hdr-ROBJ8 2026-02-22T20:44:16.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ8 key-ROBJ8 val-ROBJ8 2026-02-22T20:44:16.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:16.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ9 2026-02-22T20:44:16.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ9 2026-02-22T20:44:16.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:16.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:16.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ9 2026-02-22T20:44:16.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:16.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:16.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:17.180 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:17.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:17.385 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:17.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:17.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:17.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:17.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ9 hdr-ROBJ9 2026-02-22T20:44:17.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ9 key-ROBJ9 val-ROBJ9 2026-02-22T20:44:17.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:17.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ10 2026-02-22T20:44:17.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ10 2026-02-22T20:44:17.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:17.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:17.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ10 2026-02-22T20:44:17.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:17.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:17.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:17.688 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:17.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:17.891 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:17.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:17.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:17.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ10 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:17.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ10 hdr-ROBJ10 2026-02-22T20:44:17.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ10 key-ROBJ10 val-ROBJ10 2026-02-22T20:44:17.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:17.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ11 2026-02-22T20:44:17.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ11 2026-02-22T20:44:17.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:17.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:17.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ11 2026-02-22T20:44:17.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:17.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:17.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:18.188 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:18.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:18.393 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:18.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:18.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:18.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ11 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:18.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ11 hdr-ROBJ11 2026-02-22T20:44:18.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ11 key-ROBJ11 val-ROBJ11 2026-02-22T20:44:18.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:18.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ12 2026-02-22T20:44:18.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ12 2026-02-22T20:44:18.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:18.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:18.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ12 2026-02-22T20:44:18.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:18.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:18.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:18.682 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:18.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:18.887 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:18.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:18.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:18.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ12 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:18.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ12 hdr-ROBJ12 2026-02-22T20:44:18.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ12 key-ROBJ12 val-ROBJ12 2026-02-22T20:44:18.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:18.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ13 2026-02-22T20:44:18.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ13 2026-02-22T20:44:18.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:18.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:18.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ13 2026-02-22T20:44:18.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:18.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:18.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:19.179 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:19.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:19.384 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:19.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:19.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:19.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ13 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:19.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ13 hdr-ROBJ13 2026-02-22T20:44:19.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ13 key-ROBJ13 val-ROBJ13 2026-02-22T20:44:19.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:19.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ14 2026-02-22T20:44:19.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ14 2026-02-22T20:44:19.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:19.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:19.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ14 2026-02-22T20:44:19.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:19.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:19.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:19.687 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:19.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:19.890 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:19.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:19.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:19.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ14 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:19.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ14 hdr-ROBJ14 2026-02-22T20:44:19.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ14 key-ROBJ14 val-ROBJ14 2026-02-22T20:44:19.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:19.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ15 2026-02-22T20:44:19.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ15 2026-02-22T20:44:19.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:19.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:19.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ15 2026-02-22T20:44:19.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:19.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:19.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:20.182 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:20.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:20.388 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:20.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:20.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:20.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ15 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:20.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ15 hdr-ROBJ15 2026-02-22T20:44:20.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ15 key-ROBJ15 val-ROBJ15 2026-02-22T20:44:20.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:20.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ16 2026-02-22T20:44:20.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ16 2026-02-22T20:44:20.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:20.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:20.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ16 2026-02-22T20:44:20.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:20.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:20.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:20.682 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:20.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:20.887 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:20.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:20.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:20.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ16 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:20.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ16 hdr-ROBJ16 2026-02-22T20:44:20.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ16 key-ROBJ16 val-ROBJ16 2026-02-22T20:44:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ17 2026-02-22T20:44:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ17 2026-02-22T20:44:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ17 2026-02-22T20:44:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:20.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:21.182 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:21.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:21.388 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:21.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:21.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:21.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ17 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:21.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ17 hdr-ROBJ17 2026-02-22T20:44:21.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ17 key-ROBJ17 val-ROBJ17 2026-02-22T20:44:21.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:21.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ18 2026-02-22T20:44:21.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ18 2026-02-22T20:44:21.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:21.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:21.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ18 2026-02-22T20:44:21.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:21.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:21.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:21.691 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:21.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:21.894 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:21.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:21.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:21.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ18 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:21.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ18 hdr-ROBJ18 2026-02-22T20:44:21.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ18 key-ROBJ18 val-ROBJ18 2026-02-22T20:44:21.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6457: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:21.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6458: TEST_dual_store_replicated_cluster: objname=ROBJ19 2026-02-22T20:44:21.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6459: TEST_dual_store_replicated_cluster: add_something td/osd-scrub-repair csr_pool ROBJ19 2026-02-22T20:44:21.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:44:21.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=csr_pool 2026-02-22T20:44:21.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=ROBJ19 2026-02-22T20:44:21.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:44:21.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:44:21.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:44:22.187 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:44:22.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:44:22.392 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:44:22.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:44:22.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:44:22.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:44:22.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6461: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapheader ROBJ19 hdr-ROBJ19 2026-02-22T20:44:22.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6462: TEST_dual_store_replicated_cluster: rados --pool csr_pool setomapval ROBJ19 key-ROBJ19 val-ROBJ19 2026-02-22T20:44:22.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6466: TEST_dual_store_replicated_cluster: dd if=/dev/zero of=td/osd-scrub-repair/new.ROBJ19 bs=1024 count=1025 2026-02-22T20:44:22.469 INFO:tasks.workunit.client.0.trial150.stderr:1025+0 records in 2026-02-22T20:44:22.469 INFO:tasks.workunit.client.0.trial150.stderr:1025+0 records out 2026-02-22T20:44:22.469 INFO:tasks.workunit.client.0.trial150.stderr:1049600 bytes (1.0 MB, 1.0 MiB) copied, 0.00229406 s, 458 MB/s 2026-02-22T20:44:22.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6467: TEST_dual_store_replicated_cluster: rados --pool csr_pool put ROBJ19 td/osd-scrub-repair/new.ROBJ19 2026-02-22T20:44:22.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6468: TEST_dual_store_replicated_cluster: rm -f td/osd-scrub-repair/new.ROBJ19 2026-02-22T20:44:22.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6470: TEST_dual_store_replicated_cluster: get_pg csr_pool ROBJ0 2026-02-22T20:44:22.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-22T20:44:22.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-22T20:44:22.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-22T20:44:22.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:44:22.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6470: TEST_dual_store_replicated_cluster: local pg=3.0 2026-02-22T20:44:22.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6471: TEST_dual_store_replicated_cluster: get_primary csr_pool ROBJ0 2026-02-22T20:44:22.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=csr_pool 2026-02-22T20:44:22.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=ROBJ0 2026-02-22T20:44:22.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map csr_pool ROBJ0 2026-02-22T20:44:22.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:44:22.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6471: TEST_dual_store_replicated_cluster: local primary=1 2026-02-22T20:44:22.810 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6474: TEST_dual_store_replicated_cluster: get_asok_path osd.0 2026-02-22T20:44:22.810 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:44:22.810 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:44:22.810 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:44:22.810 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:44:22.810 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:44:22.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:44:22.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6474: TEST_dual_store_replicated_cluster: CEPH_ARGS= 2026-02-22T20:44:22.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6474: TEST_dual_store_replicated_cluster: ceph daemon /tmp/ceph-asok.40073/ceph-osd.0.asok config set osd_deep_scrub_update_digest_min_age 0 2026-02-22T20:44:22.874 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:44:22.875 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2026-02-22T20:44:22.875 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:44:22.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6476: TEST_dual_store_replicated_cluster: get_asok_path osd.1 2026-02-22T20:44:22.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:44:22.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:44:22.881 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:44:22.881 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:44:22.881 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:44:22.882 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:44:22.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6476: TEST_dual_store_replicated_cluster: CEPH_ARGS= 2026-02-22T20:44:22.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6476: TEST_dual_store_replicated_cluster: ceph daemon /tmp/ceph-asok.40073/ceph-osd.1.asok config set osd_deep_scrub_update_digest_min_age 0 2026-02-22T20:44:22.935 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:44:22.935 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_deep_scrub_update_digest_min_age = '' (not observed, change may require restart) " 2026-02-22T20:44:22.935 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:44:22.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6478: TEST_dual_store_replicated_cluster: pg_deep_scrub 3.0 2026-02-22T20:44:22.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-22T20:44:22.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-22T20:44:22.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:44:22.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:44:22.942 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:44:22.942 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:44:22.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:44:22.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:44:22.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:44:23.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:44:23.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:44:23.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:44:23.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:44:23.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:44:23.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:44:23.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:44:23.397 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:44:23.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:44:23.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:23.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:44:23.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836505 2026-02-22T20:44:23.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836505 2026-02-22T20:44:23.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505' 2026-02-22T20:44:23.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:23.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:44:23.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705687 2026-02-22T20:44:23.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705687 2026-02-22T20:44:23.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836505 1-38654705687' 2026-02-22T20:44:23.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:23.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836505 2026-02-22T20:44:23.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:23.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:44:23.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836505 2026-02-22T20:44:23.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:23.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836505 2026-02-22T20:44:23.549 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836505 2026-02-22T20:44:23.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2026-02-22T20:44:23.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:23.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836504 -lt 21474836505 2026-02-22T20:44:23.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:24.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:44:24.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:24.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836504 -lt 21474836505 2026-02-22T20:44:24.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:25.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:44:25.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:26.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836507 -lt 21474836505 2026-02-22T20:44:26.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:26.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705687 2026-02-22T20:44:26.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:26.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:44:26.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705687 2026-02-22T20:44:26.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:26.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705687 2026-02-22T20:44:26.042 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705687 2026-02-22T20:44:26.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705687' 2026-02-22T20:44:26.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:44:26.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705688 -lt 38654705687 2026-02-22T20:44:26.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:44:26.199 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:44:26.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:44:26.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:44:26.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:44:26.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:44:26.200 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:44:26.200 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:44:26.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T20:44:26.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:44:26.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:44:26.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:44:26.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:44:26.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:44:26.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:44:26.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:44:26.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:44:26.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-22T20:44:09.105333+0000 2026-02-22T20:44:26.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-22T20:44:26.584 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-22T20:44:26.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-22T20:44:09.105333+0000 last_deep_scrub_stamp 2026-02-22T20:44:26.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:44:26.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:44:09.105333+0000 2026-02-22T20:44:26.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-22T20:44:26.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:44:26.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:44:26.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:44:26.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:44:26.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:44:26.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:44:26.597 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:44:26.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:44:09.105333+0000 '>' 2026-02-22T20:44:09.105333+0000 2026-02-22T20:44:26.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:44:27.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:44:27.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:44:27.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:44:27.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:44:27.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:44:27.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:44:27.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:44:27.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:44:09.105333+0000 '>' 2026-02-22T20:44:09.105333+0000 2026-02-22T20:44:27.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:44:28.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:44:28.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:44:28.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:44:28.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:44:28.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:44:28.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:44:28.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:44:29.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:44:27.072684+0000 '>' 2026-02-22T20:44:09.105333+0000 2026-02-22T20:44:29.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:44:29.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: seq 1 19 2026-02-22T20:44:29.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:29.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ1 2026-02-22T20:44:29.069 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 1 % 2 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6489: TEST_dual_store_replicated_cluster: local payload=UVWXYZZZ 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6490: TEST_dual_store_replicated_cluster: echo UVWXYZZZ 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6491: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:44:29.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:44:29.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:44:29.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:44:29.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:44:29.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:44:29.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:44:29.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:44:29.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:44:29.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:44:29.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:44:29.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:44:29.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:44:29.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:44:29.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ1 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:44:29.580 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#-5:00000000:::deep_scrub_3.0:head#, (61) No data available 2026-02-22T20:44:29.581 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#-5:00000000:::scrub_3.0:head#, (61) No data available 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:44:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:44:29.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:44:29.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:44:29.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:44:29.679 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:44:29.679 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:44:29.679 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:44:29.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:44:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:44:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:44:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:44:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:44:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:44:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:44:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:44:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:44:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:44:29.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:44:29.681 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:44:29.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:29.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:44:29.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:44:29.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:44:29.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:44:29.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:44:29.698 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:29.684+0000 7f629260ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:29.699 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:29.684+0000 7f629260ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:29.700 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:29.684+0000 7f629260ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:29.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:44:29.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:44:29.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:44:29.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:44:29.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:44:29.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:29.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:44:29.843 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:44:29.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:29.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:44:30.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:44:30.035 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:30.020+0000 7f629260ca40 -1 Falling back to public interface 2026-02-22T20:44:30.602 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:30.588+0000 7f629260ca40 -1 osd.1 63 log_to_monitors true 2026-02-22T20:44:31.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:44:31.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:31.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:44:31.002 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:44:31.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:31.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:44:31.159 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 67 up_thru 67 down_at 64 last_clean_interval [9,63) [v2:127.0.0.1:6810/3516577260,v1:127.0.0.1:6811/3516577260] [v2:127.0.0.1:6812/3516577260,v1:127.0.0.1:6813/3516577260] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:44:31.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:44:31.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:44:31.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:44:31.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:44:31.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:44:31.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:44:31.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:44:31.160 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:44:31.160 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:44:31.160 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:44:31.160 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:44:31.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:44:31.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:44:31.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:44:31.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:44:31.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:44:31.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:44:31.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:44:31.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:44:31.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:44:31.433 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:44:31.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:44:31.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:31.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:44:31.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836514 2026-02-22T20:44:31.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836514 2026-02-22T20:44:31.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514' 2026-02-22T20:44:31.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:31.508 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:44:31.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808834 2026-02-22T20:44:31.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808834 2026-02-22T20:44:31.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836514 1-287762808834' 2026-02-22T20:44:31.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:31.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836514 2026-02-22T20:44:31.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:31.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:44:31.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836514 2026-02-22T20:44:31.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:31.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836514 2026-02-22T20:44:31.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836514' 2026-02-22T20:44:31.585 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836514 2026-02-22T20:44:31.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:31.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836514 2026-02-22T20:44:31.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:32.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:44:32.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:32.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836514 2026-02-22T20:44:32.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:33.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:44:33.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:34.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836514 2026-02-22T20:44:34.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:34.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-287762808834 2026-02-22T20:44:34.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:34.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:44:34.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-287762808834 2026-02-22T20:44:34.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:34.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808834 2026-02-22T20:44:34.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 287762808834' 2026-02-22T20:44:34.072 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 287762808834 2026-02-22T20:44:34.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:44:34.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808836 -lt 287762808834 2026-02-22T20:44:34.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:44:34.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:34.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:34.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:44:34.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:44:34.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:44:34.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:44:34.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:44:34.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:44:34.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:44:34.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:44:34.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:44:34.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:44:34.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:34.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:34.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:44:34.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:44:34.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:44:34.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:34.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ2 2026-02-22T20:44:34.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 2 % 2 2026-02-22T20:44:34.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-22T20:44:34.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:44:34.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6496: TEST_dual_store_replicated_cluster: local payload=UVWXYZ 2026-02-22T20:44:34.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6497: TEST_dual_store_replicated_cluster: echo UVWXYZ 2026-02-22T20:44:34.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6498: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:44:34.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:44:34.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:44:34.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:44:34.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:44:34.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:44:34.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:44:34.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:44:34.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:44:34.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:44:34.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:44:34.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:44:34.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:44:34.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:44:34.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:44:34.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:44:35.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:44:35.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:44:35.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:44:35.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:44:35.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:44:35.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:44:35.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:44:35.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ2 set-bytes td/osd-scrub-repair/CORRUPT 2026-02-22T20:44:35.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:44:35.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:44:35.404 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:44:35.404 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:44:35.404 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:44:35.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:35.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:35.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:44:35.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:44:35.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:44:35.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:44:35.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:44:35.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:44:35.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:44:35.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:44:35.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:44:35.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:44:35.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:44:35.405 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:44:35.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:35.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:44:35.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:44:35.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:44:35.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:44:35.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:44:35.424 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:35.408+0000 7f4775584a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:35.425 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:35.412+0000 7f4775584a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:35.426 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:35.412+0000 7f4775584a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:35.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:44:35.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:44:35.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:44:35.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:44:35.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:44:35.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:35.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:44:35.570 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:44:35.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:35.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:44:35.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:44:35.847 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:35.832+0000 7f4775584a40 -1 Falling back to public interface 2026-02-22T20:44:36.398 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:36.384+0000 7f4775584a40 -1 osd.0 69 log_to_monitors true 2026-02-22T20:44:36.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:44:36.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:36.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:44:36.729 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:44:36.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:36.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:44:36.885 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 73 up_thru 73 down_at 70 last_clean_interval [5,69) [v2:127.0.0.1:6802/2227372569,v1:127.0.0.1:6803/2227372569] [v2:127.0.0.1:6804/2227372569,v1:127.0.0.1:6805/2227372569] exists,up 976f5525-b691-48b1-b2d2-8b61723446d8 2026-02-22T20:44:36.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:44:36.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:44:36.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:44:36.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:44:36.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:44:36.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:44:36.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:44:36.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:44:36.886 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:44:36.886 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:44:36.886 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:44:36.886 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:44:36.886 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:44:37.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:44:37.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:44:37.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:44:37.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:44:37.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:44:37.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:44:37.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:44:37.162 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:44:37.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:44:37.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:37.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:44:37.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612610 2026-02-22T20:44:37.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612610 2026-02-22T20:44:37.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-313532612610' 2026-02-22T20:44:37.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:37.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:44:37.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=287762808841 2026-02-22T20:44:37.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 287762808841 2026-02-22T20:44:37.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-313532612610 1-287762808841' 2026-02-22T20:44:37.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:37.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-313532612610 2026-02-22T20:44:37.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:37.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:44:37.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-313532612610 2026-02-22T20:44:37.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:37.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612610 2026-02-22T20:44:37.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 313532612610' 2026-02-22T20:44:37.313 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 313532612610 2026-02-22T20:44:37.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:37.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 313532612610 2026-02-22T20:44:37.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:38.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:44:38.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:38.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 313532612610 2026-02-22T20:44:38.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:39.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:44:39.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:39.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612612 -lt 313532612610 2026-02-22T20:44:39.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:39.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-287762808841 2026-02-22T20:44:39.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:39.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:44:39.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-287762808841 2026-02-22T20:44:39.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:39.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=287762808841 2026-02-22T20:44:39.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 287762808841' 2026-02-22T20:44:39.813 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 287762808841 2026-02-22T20:44:39.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:44:39.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 287762808843 -lt 287762808841 2026-02-22T20:44:39.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:44:39.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:39.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:40.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:44:40.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:44:40.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:44:40.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:44:40.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:44:40.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:44:40.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:44:40.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:44:40.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:44:40.343 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:44:40.343 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:40.343 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:40.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:44:40.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:44:40.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:44:40.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:40.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ3 2026-02-22T20:44:40.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 3 % 2 2026-02-22T20:44:40.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-22T20:44:40.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:44:40.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6503: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ3 remove 2026-02-22T20:44:40.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:44:40.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:44:40.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:44:40.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:44:40.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ3 remove 2026-02-22T20:44:40.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:44:40.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:44:40.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:44:40.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:44:40.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:44:40.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:44:40.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:44:40.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:44:40.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:44:40.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:44:40.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:44:40.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ3 remove 2026-02-22T20:44:40.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:44:40.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:44:40.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:44:40.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:44:40.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:44:40.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ3 remove 2026-02-22T20:44:41.026 INFO:tasks.workunit.client.0.trial150.stdout:remove #3:f2a5b2a4:::ROBJ3:head# 2026-02-22T20:44:41.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:41.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:44:41.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:44:41.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:44:41.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:44:41.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:44:41.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:44:41.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:44:41.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:44:41.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:44:41.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:44:41.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:44:41.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:44:41.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:44:41.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:44:41.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:44:41.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:44:41.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:44:41.129 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:44:41.129 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:44:41.129 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:44:41.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:41.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:41.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:44:41.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:44:41.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:44:41.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:44:41.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:44:41.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:44:41.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:44:41.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:44:41.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:44:41.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:44:41.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:44:41.131 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:44:41.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:41.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:44:41.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:44:41.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:44:41.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:44:41.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:44:41.148 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:41.132+0000 7fec85b74a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:41.150 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:41.136+0000 7fec85b74a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:41.151 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:41.136+0000 7fec85b74a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:41.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:44:41.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:44:41.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:44:41.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:44:41.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:44:41.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:41.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:44:41.290 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:44:41.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:41.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:44:41.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:44:41.567 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:41.552+0000 7fec85b74a40 -1 Falling back to public interface 2026-02-22T20:44:42.196 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:42.180+0000 7fec85b74a40 -1 osd.1 75 log_to_monitors true 2026-02-22T20:44:42.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:44:42.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:42.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:44:42.452 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:44:42.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:42.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:44:42.613 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 79 up_thru 79 down_at 76 last_clean_interval [67,75) [v2:127.0.0.1:6810/4018548336,v1:127.0.0.1:6811/4018548336] [v2:127.0.0.1:6812/4018548336,v1:127.0.0.1:6813/4018548336] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:44:42.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:44:42.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:44:42.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:44:42.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:44:42.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:44:42.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:44:42.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:44:42.613 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:44:42.614 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:44:42.614 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:44:42.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:44:42.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:44:42.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:44:42.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:44:42.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:44:42.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:44:42.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:44:42.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:44:42.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:44:42.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:44:42.893 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:44:42.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:44:42.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:42.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:44:42.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=313532612617 2026-02-22T20:44:42.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 313532612617 2026-02-22T20:44:42.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-313532612617' 2026-02-22T20:44:42.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:42.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:44:43.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=339302416386 2026-02-22T20:44:43.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 339302416386 2026-02-22T20:44:43.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-313532612617 1-339302416386' 2026-02-22T20:44:43.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:43.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-313532612617 2026-02-22T20:44:43.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:43.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:44:43.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-313532612617 2026-02-22T20:44:43.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:43.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=313532612617 2026-02-22T20:44:43.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 313532612617' 2026-02-22T20:44:43.049 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 313532612617 2026-02-22T20:44:43.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:43.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612617 -lt 313532612617 2026-02-22T20:44:43.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:43.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-339302416386 2026-02-22T20:44:43.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:43.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:44:43.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-339302416386 2026-02-22T20:44:43.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:43.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=339302416386 2026-02-22T20:44:43.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 339302416386' 2026-02-22T20:44:43.214 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 339302416386 2026-02-22T20:44:43.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:44:43.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 339302416386 2026-02-22T20:44:43.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:44.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:44:44.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:44:44.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 339302416386 2026-02-22T20:44:44.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:45.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:44:45.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:44:45.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 339302416388 -lt 339302416386 2026-02-22T20:44:45.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:44:45.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:45.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:45.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:44:45.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:44:45.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:44:45.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:44:45.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:44:45.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:44:45.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:44:45.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:44:46.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:44:46.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:44:46.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:46.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:46.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:44:46.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:44:46.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:44:46.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:46.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ4 2026-02-22T20:44:46.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 4 % 2 2026-02-22T20:44:46.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-22T20:44:46.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:44:46.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6508: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-22T20:44:46.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:44:46.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:44:46.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:44:46.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:44:46.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-22T20:44:46.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:44:46.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:44:46.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:44:46.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:44:46.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:44:46.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:44:46.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:44:46.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:44:46.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:44:46.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:44:46.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:44:46.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-22T20:44:46.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:44:46.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:44:46.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:44:46.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:44:46.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:44:46.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ4 set-omap key-ROBJ4 td/osd-scrub-repair/CORRUPT 2026-02-22T20:44:46.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:46.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:44:46.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:44:46.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:44:46.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:44:46.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:44:46.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:44:46.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:44:46.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:44:46.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:44:46.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:44:46.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:44:46.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:44:46.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:44:46.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:44:46.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:44:46.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:44:46.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:44:46.829 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:44:46.829 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:44:46.829 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:44:46.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:46.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:46.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:44:46.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:44:46.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:44:46.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:44:46.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:44:46.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:44:46.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:44:46.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:44:46.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:44:46.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:44:46.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:44:46.832 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:44:46.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:46.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:44:46.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:44:46.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:44:46.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:44:46.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:44:46.856 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:46.840+0000 7fedc9c01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:46.857 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:46.840+0000 7fedc9c01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:46.858 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:46.844+0000 7fedc9c01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:46.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:44:46.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:44:46.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:44:46.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:44:46.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:44:46.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:46.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:44:46.994 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:44:46.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:46.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:44:47.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:44:47.255 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:47.240+0000 7fedc9c01a40 -1 Falling back to public interface 2026-02-22T20:44:47.821 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:47.804+0000 7fedc9c01a40 -1 osd.0 80 log_to_monitors true 2026-02-22T20:44:48.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:44:48.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:48.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:44:48.153 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:44:48.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:48.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:44:48.306 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 84 up_thru 84 down_at 81 last_clean_interval [73,80) [v2:127.0.0.1:6802/3173803066,v1:127.0.0.1:6803/3173803066] [v2:127.0.0.1:6804/3173803066,v1:127.0.0.1:6805/3173803066] exists,up 976f5525-b691-48b1-b2d2-8b61723446d8 2026-02-22T20:44:48.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:44:48.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:44:48.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:44:48.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:44:48.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:44:48.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:44:48.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:44:48.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:44:48.308 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:44:48.308 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:44:48.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:44:48.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:44:48.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:44:48.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:44:48.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:44:48.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:44:48.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:44:48.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:44:48.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:44:48.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:44:48.588 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:44:48.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:44:48.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:48.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:44:48.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=360777252866 2026-02-22T20:44:48.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 360777252866 2026-02-22T20:44:48.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866' 2026-02-22T20:44:48.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:48.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:44:48.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=339302416393 2026-02-22T20:44:48.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 339302416393 2026-02-22T20:44:48.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252866 1-339302416393' 2026-02-22T20:44:48.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:48.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-360777252866 2026-02-22T20:44:48.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:48.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:44:48.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-360777252866 2026-02-22T20:44:48.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:48.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=360777252866 2026-02-22T20:44:48.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 360777252866' 2026-02-22T20:44:48.737 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 360777252866 2026-02-22T20:44:48.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:48.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 313532612620 -lt 360777252866 2026-02-22T20:44:48.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:49.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:44:49.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:50.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252867 -lt 360777252866 2026-02-22T20:44:50.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:50.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-339302416393 2026-02-22T20:44:50.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:50.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:44:50.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-339302416393 2026-02-22T20:44:50.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:50.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=339302416393 2026-02-22T20:44:50.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 339302416393' 2026-02-22T20:44:50.069 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 339302416393 2026-02-22T20:44:50.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:44:50.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 339302416393 -lt 339302416393 2026-02-22T20:44:50.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:44:50.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:50.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:50.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:44:50.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:44:50.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:44:50.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:44:50.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:44:50.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:44:50.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:44:50.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:44:50.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:44:50.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:44:50.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:50.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:50.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:44:50.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:44:50.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:44:50.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:50.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ5 2026-02-22T20:44:50.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 5 % 2 2026-02-22T20:44:50.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-22T20:44:50.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:44:50.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6513: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-22T20:44:50.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:44:50.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:44:50.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:44:50.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:44:50.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-22T20:44:50.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:44:50.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:44:50.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:44:50.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:44:50.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:44:50.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:44:50.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:44:50.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:44:50.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:44:50.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:44:51.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:44:51.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ5 rm-omap key-ROBJ5 2026-02-22T20:44:51.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:44:51.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:44:51.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:44:51.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:44:51.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:44:51.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ5 rm-omap key-ROBJ5 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:44:51.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:44:51.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:44:51.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:44:51.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:44:51.363 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:44:51.363 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:44:51.363 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:44:51.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:44:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:44:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:44:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:44:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:44:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:44:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:44:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:44:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:44:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:44:51.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:44:51.365 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:44:51.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:51.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:44:51.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:44:51.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:44:51.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:44:51.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:44:51.387 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:51.372+0000 7f0cc4f93a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:51.388 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:51.372+0000 7f0cc4f93a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:51.389 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:51.372+0000 7f0cc4f93a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:51.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:44:51.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:44:51.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:44:51.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:44:51.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:44:51.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:51.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:44:51.529 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:44:51.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:51.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:44:51.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:44:51.783 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:51.768+0000 7f0cc4f93a40 -1 Falling back to public interface 2026-02-22T20:44:52.430 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:52.416+0000 7f0cc4f93a40 -1 osd.1 86 log_to_monitors true 2026-02-22T20:44:52.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:44:52.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:52.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:44:52.692 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:44:52.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:52.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:44:52.848 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 90 up_thru 90 down_at 87 last_clean_interval [79,86) [v2:127.0.0.1:6810/768406802,v1:127.0.0.1:6811/768406802] [v2:127.0.0.1:6812/768406802,v1:127.0.0.1:6813/768406802] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:44:52.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:44:52.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:44:52.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:44:52.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:44:52.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:44:52.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:44:52.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:44:52.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:44:52.850 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:44:52.850 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:44:52.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:44:52.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:44:52.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:44:52.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:44:52.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:44:52.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:44:52.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:44:52.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:44:52.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:44:53.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:44:53.131 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:44:53.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:44:53.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:53.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:44:53.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=360777252872 2026-02-22T20:44:53.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 360777252872 2026-02-22T20:44:53.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252872' 2026-02-22T20:44:53.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:53.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:44:53.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056642 2026-02-22T20:44:53.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056642 2026-02-22T20:44:53.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-360777252872 1-386547056642' 2026-02-22T20:44:53.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:53.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-360777252872 2026-02-22T20:44:53.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:53.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:44:53.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-360777252872 2026-02-22T20:44:53.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:53.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=360777252872 2026-02-22T20:44:53.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 360777252872' 2026-02-22T20:44:53.283 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 360777252872 2026-02-22T20:44:53.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:53.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252871 -lt 360777252872 2026-02-22T20:44:53.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:54.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:44:54.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:54.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252871 -lt 360777252872 2026-02-22T20:44:54.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:44:55.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:44:55.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:55.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 360777252874 -lt 360777252872 2026-02-22T20:44:55.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:55.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-386547056642 2026-02-22T20:44:55.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:55.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:44:55.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-386547056642 2026-02-22T20:44:55.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:55.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056642 2026-02-22T20:44:55.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 386547056642' 2026-02-22T20:44:55.775 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 386547056642 2026-02-22T20:44:55.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:44:55.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056644 -lt 386547056642 2026-02-22T20:44:55.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:44:55.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:55.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:56.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:44:56.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:44:56.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:44:56.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:44:56.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:44:56.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:44:56.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:44:56.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:44:56.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:44:56.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:44:56.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:44:56.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:44:56.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:44:56.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:44:56.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:44:56.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:44:56.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ6 2026-02-22T20:44:56.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 6 % 2 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6518: TEST_dual_store_replicated_cluster: echo extra 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6519: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:44:56.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:44:56.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:44:56.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:44:56.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:44:56.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:44:56.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:44:56.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:44:56.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-22T20:44:56.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:44:56.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:44:56.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:44:56.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:44:56.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:44:56.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ6 set-omap key2-ROBJ6 td/osd-scrub-repair/extra-val 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:44:57.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:44:57.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:44:57.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:44:57.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:44:57.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:44:57.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:44:57.083 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:44:57.083 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:44:57.083 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:44:57.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:57.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:44:57.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:44:57.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:44:57.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:44:57.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:44:57.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:44:57.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:44:57.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:44:57.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:44:57.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:44:57.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:44:57.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:44:57.085 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:44:57.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:44:57.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:44:57.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:44:57.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:44:57.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:44:57.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:44:57.107 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:57.092+0000 7f12dcee8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:57.109 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:57.092+0000 7f12dcee8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:57.110 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:57.096+0000 7f12dcee8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:44:57.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:44:57.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:44:57.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:44:57.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:44:57.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:44:57.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:57.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:44:57.249 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:44:57.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:57.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:44:57.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:44:57.503 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:57.488+0000 7f12dcee8a40 -1 Falling back to public interface 2026-02-22T20:44:58.134 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:44:58.120+0000 7f12dcee8a40 -1 osd.0 91 log_to_monitors true 2026-02-22T20:44:58.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:44:58.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:44:58.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:44:58.410 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:44:58.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:44:58.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:44:58.567 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 95 up_thru 95 down_at 92 last_clean_interval [84,91) [v2:127.0.0.1:6802/1471509790,v1:127.0.0.1:6803/1471509790] [v2:127.0.0.1:6804/1471509790,v1:127.0.0.1:6805/1471509790] exists,up 976f5525-b691-48b1-b2d2-8b61723446d8 2026-02-22T20:44:58.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:44:58.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:44:58.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:44:58.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:44:58.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:44:58.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:44:58.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:44:58.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:44:58.569 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:44:58.569 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:44:58.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:44:58.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:44:58.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:44:58.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:44:58.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:44:58.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:44:58.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:44:58.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:44:58.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:44:58.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:44:58.847 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:44:58.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:44:58.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:58.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:44:58.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893122 2026-02-22T20:44:58.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893122 2026-02-22T20:44:58.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-408021893122' 2026-02-22T20:44:58.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:44:58.924 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:44:58.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=386547056649 2026-02-22T20:44:58.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 386547056649 2026-02-22T20:44:58.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-408021893122 1-386547056649' 2026-02-22T20:44:58.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:58.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-408021893122 2026-02-22T20:44:58.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:58.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:44:58.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-408021893122 2026-02-22T20:44:58.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:58.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893122 2026-02-22T20:44:58.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 408021893122' 2026-02-22T20:44:58.999 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 408021893122 2026-02-22T20:44:58.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:44:59.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893122 -lt 408021893122 2026-02-22T20:44:59.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:44:59.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-386547056649 2026-02-22T20:44:59.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:44:59.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:44:59.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-386547056649 2026-02-22T20:44:59.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:44:59.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=386547056649 2026-02-22T20:44:59.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 386547056649' 2026-02-22T20:44:59.167 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 386547056649 2026-02-22T20:44:59.167 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:44:59.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056648 -lt 386547056649 2026-02-22T20:44:59.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:00.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:45:00.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:00.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056648 -lt 386547056649 2026-02-22T20:45:00.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:01.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:45:01.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:01.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 386547056651 -lt 386547056649 2026-02-22T20:45:01.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:45:01.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:01.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:01.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:45:01.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:45:01.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:45:01.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:45:01.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:45:01.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:45:01.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:45:01.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:45:02.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:45:02.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:45:02.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:02.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:02.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:45:02.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:45:02.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:45:02.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6520: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/extra-val 2026-02-22T20:45:02.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:45:02.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ7 2026-02-22T20:45:02.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 7 % 2 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6525: TEST_dual_store_replicated_cluster: echo -n newheader 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6526: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:45:02.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:45:02.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:45:02.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:45:02.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:45:02.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:45:02.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:45:02.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:45:02.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-22T20:45:02.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:45:02.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:45:02.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:45:02.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:45:02.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:02.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ7 set-omaphdr td/osd-scrub-repair/hdr 2026-02-22T20:45:02.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:02.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:45:02.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:45:02.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:45:02.793 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:45:02.794 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:45:02.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:02.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:02.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:45:02.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:45:02.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:45:02.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:45:02.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:45:02.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:45:02.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:45:02.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:45:02.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:45:02.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:45:02.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:45:02.796 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:45:02.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:02.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:45:02.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:45:02.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:45:02.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:45:02.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:45:02.815 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:02.801+0000 7f4b85ddca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:02.816 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:02.801+0000 7f4b85ddca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:02.817 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:02.805+0000 7f4b85ddca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:02.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:45:02.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:45:02.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:45:02.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:45:02.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:45:02.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:02.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:45:02.954 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:45:02.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:02.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:03.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:45:03.163 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:03.149+0000 7f4b85ddca40 -1 Falling back to public interface 2026-02-22T20:45:03.773 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:03.757+0000 7f4b85ddca40 -1 osd.1 97 log_to_monitors true 2026-02-22T20:45:04.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:45:04.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:04.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:45:04.115 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:45:04.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:04.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:04.275 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 101 up_thru 101 down_at 98 last_clean_interval [90,97) [v2:127.0.0.1:6810/4198135764,v1:127.0.0.1:6811/4198135764] [v2:127.0.0.1:6812/4198135764,v1:127.0.0.1:6813/4198135764] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:45:04.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:45:04.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:45:04.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:45:04.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:45:04.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:45:04.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:45:04.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:45:04.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:45:04.277 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:45:04.277 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:45:04.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:45:04.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:45:04.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:45:04.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:45:04.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:45:04.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:45:04.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:45:04.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:45:04.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:45:04.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:45:04.549 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:45:04.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:45:04.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:04.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:45:04.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=408021893129 2026-02-22T20:45:04.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 408021893129 2026-02-22T20:45:04.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-408021893129' 2026-02-22T20:45:04.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:04.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:45:04.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696898 2026-02-22T20:45:04.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696898 2026-02-22T20:45:04.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-408021893129 1-433791696898' 2026-02-22T20:45:04.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:04.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-408021893129 2026-02-22T20:45:04.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:04.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:45:04.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-408021893129 2026-02-22T20:45:04.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:04.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=408021893129 2026-02-22T20:45:04.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 408021893129' 2026-02-22T20:45:04.697 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 408021893129 2026-02-22T20:45:04.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:04.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893126 -lt 408021893129 2026-02-22T20:45:04.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:05.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:45:05.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 408021893129 -lt 408021893129 2026-02-22T20:45:06.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:06.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-433791696898 2026-02-22T20:45:06.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:06.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:45:06.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-433791696898 2026-02-22T20:45:06.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:06.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696898 2026-02-22T20:45:06.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 433791696898' 2026-02-22T20:45:06.020 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 433791696898 2026-02-22T20:45:06.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:06.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696899 -lt 433791696898 2026-02-22T20:45:06.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:45:06.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:06.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:06.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:45:06.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:45:06.395 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:45:06.395 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:45:06.395 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:45:06.395 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:45:06.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:45:06.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:45:06.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:45:06.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:45:06.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:06.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:06.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:45:06.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:45:06.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:45:06.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6527: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/hdr 2026-02-22T20:45:06.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:45:06.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ8 2026-02-22T20:45:06.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 8 % 2 2026-02-22T20:45:06.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-22T20:45:06.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:45:06.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6531: TEST_dual_store_replicated_cluster: rados --pool csr_pool setxattr ROBJ8 key1-ROBJ8 val1-ROBJ8 2026-02-22T20:45:06.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6532: TEST_dual_store_replicated_cluster: rados --pool csr_pool setxattr ROBJ8 key2-ROBJ8 val2-ROBJ8 2026-02-22T20:45:06.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6535: TEST_dual_store_replicated_cluster: echo -n bad-val 2026-02-22T20:45:06.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6536: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-22T20:45:06.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:45:06.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:45:06.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:45:06.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:45:06.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-22T20:45:06.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:45:06.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:45:06.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:45:06.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:45:06.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:45:06.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:45:06.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:45:06.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:45:06.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:45:06.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:45:07.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:45:07.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-22T20:45:07.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:45:07.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:45:07.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:45:07.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:45:07.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:45:07.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ8 set-attr _key1-ROBJ8 td/osd-scrub-repair/bad-val 2026-02-22T20:45:07.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:07.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:45:07.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:45:07.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:45:07.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:45:07.394 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:45:07.394 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:45:07.394 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:45:07.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:07.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:07.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:45:07.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:45:07.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:45:07.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:45:07.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:45:07.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:45:07.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:45:07.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:45:07.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:45:07.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:45:07.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:45:07.396 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:45:07.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:07.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:45:07.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:45:07.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:45:07.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:45:07.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:45:07.421 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:07.405+0000 7f2fa7e91a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:07.422 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:07.409+0000 7f2fa7e91a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:07.423 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:07.409+0000 7f2fa7e91a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:07.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:45:07.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:45:07.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:45:07.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:45:07.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:45:07.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:07.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:45:07.555 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:45:07.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:07.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:45:07.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:45:07.843 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:07.829+0000 7f2fa7e91a40 -1 Falling back to public interface 2026-02-22T20:45:08.387 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:08.373+0000 7f2fa7e91a40 -1 osd.0 102 log_to_monitors true 2026-02-22T20:45:08.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:45:08.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:08.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:45:08.720 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:45:08.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:08.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:45:08.882 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 106 up_thru 106 down_at 103 last_clean_interval [95,102) [v2:127.0.0.1:6802/534648659,v1:127.0.0.1:6803/534648659] [v2:127.0.0.1:6804/534648659,v1:127.0.0.1:6805/534648659] exists,up 976f5525-b691-48b1-b2d2-8b61723446d8 2026-02-22T20:45:08.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:45:08.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:45:08.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:45:08.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:45:08.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:45:08.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:45:08.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:45:08.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:45:08.883 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:45:08.883 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:45:08.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:45:08.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:45:08.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:45:08.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:45:08.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:45:08.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:45:08.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:45:08.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:45:08.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:45:09.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:45:09.158 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:45:09.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:45:09.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:09.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:45:09.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533378 2026-02-22T20:45:09.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533378 2026-02-22T20:45:09.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533378' 2026-02-22T20:45:09.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:09.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:45:09.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=433791696904 2026-02-22T20:45:09.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 433791696904 2026-02-22T20:45:09.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533378 1-433791696904' 2026-02-22T20:45:09.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:09.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-455266533378 2026-02-22T20:45:09.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:09.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:45:09.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-455266533378 2026-02-22T20:45:09.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:09.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533378 2026-02-22T20:45:09.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 455266533378' 2026-02-22T20:45:09.309 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 455266533378 2026-02-22T20:45:09.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:09.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 455266533378 2026-02-22T20:45:09.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:10.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:45:10.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:10.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 455266533378 2026-02-22T20:45:10.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:11.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:45:11.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:11.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533380 -lt 455266533378 2026-02-22T20:45:11.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:11.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-433791696904 2026-02-22T20:45:11.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:11.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:45:11.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-433791696904 2026-02-22T20:45:11.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:11.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=433791696904 2026-02-22T20:45:11.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 433791696904' 2026-02-22T20:45:11.798 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 433791696904 2026-02-22T20:45:11.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:11.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 433791696906 -lt 433791696904 2026-02-22T20:45:11.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:45:11.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:11.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:12.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:45:12.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:45:12.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:45:12.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:45:12.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:45:12.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:45:12.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:45:12.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:45:12.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:45:12.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:45:12.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:12.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6537: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:45:12.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:45:12.503 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:45:12.503 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:45:12.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:45:12.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:45:12.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:45:12.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:45:12.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-22T20:45:12.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:45:12.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:45:12.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:45:12.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:45:12.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:12.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 rm-attr _key2-ROBJ8 2026-02-22T20:45:13.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:13.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:45:13.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:45:13.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:45:13.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:45:13.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:13.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:45:13.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:45:13.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:45:13.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:45:13.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:45:13.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:45:13.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:45:13.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:45:13.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:45:13.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:45:13.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:45:13.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:45:13.095 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:45:13.095 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:45:13.095 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:45:13.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:13.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:13.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:45:13.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:45:13.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:45:13.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:45:13.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:45:13.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:45:13.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:45:13.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:45:13.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:45:13.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:45:13.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:45:13.097 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:45:13.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:13.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:45:13.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:45:13.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:45:13.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:45:13.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:45:13.111 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:13.097+0000 7fed2c8c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:13.112 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:13.097+0000 7fed2c8c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:13.113 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:13.101+0000 7fed2c8c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:13.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:45:13.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:45:13.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:45:13.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:45:13.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:45:13.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:13.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:45:13.251 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:45:13.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:13.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:13.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:45:13.479 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:13.465+0000 7fed2c8c4a40 -1 Falling back to public interface 2026-02-22T20:45:14.120 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:14.105+0000 7fed2c8c4a40 -1 osd.1 108 log_to_monitors true 2026-02-22T20:45:14.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:45:14.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:14.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:45:14.414 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:45:14.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:14.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:14.568 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 112 up_thru 112 down_at 109 last_clean_interval [101,108) [v2:127.0.0.1:6810/3368456029,v1:127.0.0.1:6811/3368456029] [v2:127.0.0.1:6812/3368456029,v1:127.0.0.1:6813/3368456029] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:45:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:45:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:45:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:45:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:45:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:45:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:45:14.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:45:14.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:45:14.570 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:45:14.570 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:45:14.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:45:14.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:45:14.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:45:14.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:45:14.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:45:14.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:45:14.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:45:14.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:45:14.686 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:45:14.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:45:14.849 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:45:14.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:45:14.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:14.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:45:14.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533385 2026-02-22T20:45:14.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533385 2026-02-22T20:45:14.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533385' 2026-02-22T20:45:14.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:14.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:45:14.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=481036337154 2026-02-22T20:45:14.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 481036337154 2026-02-22T20:45:14.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533385 1-481036337154' 2026-02-22T20:45:14.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:14.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-455266533385 2026-02-22T20:45:14.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:14.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:45:14.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-455266533385 2026-02-22T20:45:14.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:14.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533385 2026-02-22T20:45:14.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 455266533385' 2026-02-22T20:45:14.996 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 455266533385 2026-02-22T20:45:14.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:15.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533385 -lt 455266533385 2026-02-22T20:45:15.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:15.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-481036337154 2026-02-22T20:45:15.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:15.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:45:15.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-481036337154 2026-02-22T20:45:15.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:15.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=481036337154 2026-02-22T20:45:15.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 481036337154' 2026-02-22T20:45:15.170 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 481036337154 2026-02-22T20:45:15.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:15.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 481036337154 2026-02-22T20:45:15.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:16.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:45:16.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:16.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 481036337154 2026-02-22T20:45:16.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:17.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:45:17.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:17.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 481036337156 -lt 481036337154 2026-02-22T20:45:17.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:45:17.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:17.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:17.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:45:17.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:45:17.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:45:17.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:45:17.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:45:17.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:45:17.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:45:17.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:45:18.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:45:18.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:45:18.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:18.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:18.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:45:18.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:45:18.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:45:18.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6538: TEST_dual_store_replicated_cluster: echo -n val3-ROBJ8 2026-02-22T20:45:18.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6539: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-22T20:45:18.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:45:18.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:45:18.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:45:18.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:45:18.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-22T20:45:18.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:45:18.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:45:18.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:45:18.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:45:18.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:45:18.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:45:18.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:45:18.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:45:18.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:45:18.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:45:18.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:45:18.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-22T20:45:18.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:45:18.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:45:18.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:45:18.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:45:18.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:18.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ8 set-attr _key3-ROBJ8 td/osd-scrub-repair/newval 2026-02-22T20:45:18.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:18.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:45:18.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:45:18.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:45:18.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:45:18.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:18.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:45:18.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:45:18.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:45:18.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:45:18.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:45:18.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:45:18.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:45:18.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:45:18.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:45:18.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:45:18.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:45:18.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:45:18.826 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:45:18.826 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:45:18.826 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:45:18.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:18.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:18.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:45:18.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:45:18.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:45:18.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:45:18.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:45:18.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:45:18.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:45:18.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:45:18.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:45:18.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:45:18.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:45:18.829 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:45:18.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:18.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:45:18.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:45:18.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:45:18.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:45:18.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:45:18.855 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:18.841+0000 7fafbfd51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:18.856 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:18.841+0000 7fafbfd51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:18.857 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:18.845+0000 7fafbfd51a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:18.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:45:18.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:45:18.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:45:18.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:45:18.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:45:18.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:18.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:45:18.990 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:45:18.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:18.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:19.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:45:19.259 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:19.245+0000 7fafbfd51a40 -1 Falling back to public interface 2026-02-22T20:45:19.922 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:19.909+0000 7fafbfd51a40 -1 osd.1 113 log_to_monitors true 2026-02-22T20:45:20.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:45:20.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:20.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:45:20.153 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:45:20.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:20.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:20.306 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 117 up_thru 117 down_at 114 last_clean_interval [112,113) [v2:127.0.0.1:6810/1586190600,v1:127.0.0.1:6811/1586190600] [v2:127.0.0.1:6812/1586190600,v1:127.0.0.1:6813/1586190600] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:45:20.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:45:20.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:45:20.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:45:20.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:45:20.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:45:20.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:45:20.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:45:20.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:45:20.308 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:45:20.308 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:45:20.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:45:20.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:45:20.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:45:20.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:45:20.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:45:20.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:45:20.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:45:20.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:45:20.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:45:20.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:45:20.583 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:45:20.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:45:20.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:20.584 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:45:20.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533392 2026-02-22T20:45:20.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533392 2026-02-22T20:45:20.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533392' 2026-02-22T20:45:20.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:20.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:45:20.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=502511173634 2026-02-22T20:45:20.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 502511173634 2026-02-22T20:45:20.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533392 1-502511173634' 2026-02-22T20:45:20.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:20.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-455266533392 2026-02-22T20:45:20.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:20.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:45:20.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-455266533392 2026-02-22T20:45:20.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:20.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533392 2026-02-22T20:45:20.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 455266533392' 2026-02-22T20:45:20.733 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 455266533392 2026-02-22T20:45:20.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:20.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533389 -lt 455266533392 2026-02-22T20:45:20.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:21.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:45:21.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:22.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533392 -lt 455266533392 2026-02-22T20:45:22.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:22.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-502511173634 2026-02-22T20:45:22.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:22.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:45:22.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-502511173634 2026-02-22T20:45:22.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:22.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=502511173634 2026-02-22T20:45:22.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 502511173634' 2026-02-22T20:45:22.067 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 502511173634 2026-02-22T20:45:22.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:22.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 502511173634 -lt 502511173634 2026-02-22T20:45:22.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:45:22.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:22.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:22.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:45:22.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:45:22.428 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:45:22.428 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:45:22.428 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:45:22.428 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:45:22.428 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:45:22.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:45:22.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:45:22.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:45:22.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:22.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:22.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:45:22.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:45:22.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:45:22.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6540: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/bad-val td/osd-scrub-repair/newval 2026-02-22T20:45:22.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:45:22.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ9 2026-02-22T20:45:22.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 9 % 2 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6544: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:45:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:45:22.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:45:22.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:45:22.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:45:22.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:45:22.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:45:23.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:45:23.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 get-attr _ 2026-02-22T20:45:23.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:45:23.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:45:23.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:45:23.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:45:23.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:23.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 get-attr _ 2026-02-22T20:45:23.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:45:23.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:45:23.258 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:45:23.258 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:45:23.258 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:45:23.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:45:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:45:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:45:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:45:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:45:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:45:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:45:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:45:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:45:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:45:23.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:45:23.260 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:45:23.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:23.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:45:23.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:45:23.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:45:23.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:45:23.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:45:23.282 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:23.269+0000 7fad574f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:23.283 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:23.269+0000 7fad574f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:23.285 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:23.273+0000 7fad574f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:23.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:45:23.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:45:23.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:45:23.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:45:23.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:45:23.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:23.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:45:23.418 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:45:23.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:23.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:23.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:45:23.715 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:23.701+0000 7fad574f4a40 -1 Falling back to public interface 2026-02-22T20:45:24.285 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:24.273+0000 7fad574f4a40 -1 osd.1 118 log_to_monitors true 2026-02-22T20:45:24.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:45:24.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:24.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:45:24.583 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:45:24.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:24.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:24.747 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 122 up_thru 122 down_at 119 last_clean_interval [117,118) [v2:127.0.0.1:6810/3184417145,v1:127.0.0.1:6811/3184417145] [v2:127.0.0.1:6812/3184417145,v1:127.0.0.1:6813/3184417145] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:45:24.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:45:24.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:45:24.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:45:24.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:45:24.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:45:24.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:45:24.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:45:24.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:45:24.748 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:45:24.748 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:45:24.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:45:24.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:45:24.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:45:24.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:45:24.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:45:24.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:45:24.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:45:24.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:45:24.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:45:25.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:45:25.033 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:45:25.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:45:25.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:25.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:45:25.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533397 2026-02-22T20:45:25.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533397 2026-02-22T20:45:25.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533397' 2026-02-22T20:45:25.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:25.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:45:25.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=523986010114 2026-02-22T20:45:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 523986010114 2026-02-22T20:45:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533397 1-523986010114' 2026-02-22T20:45:25.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:25.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-455266533397 2026-02-22T20:45:25.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:25.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:45:25.180 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-455266533397 2026-02-22T20:45:25.180 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:25.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533397 2026-02-22T20:45:25.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 455266533397' 2026-02-22T20:45:25.182 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 455266533397 2026-02-22T20:45:25.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:25.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533396 -lt 455266533397 2026-02-22T20:45:25.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:26.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:45:26.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:26.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533396 -lt 455266533397 2026-02-22T20:45:26.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:27.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:45:27.503 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:27.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533399 -lt 455266533397 2026-02-22T20:45:27.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:27.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-523986010114 2026-02-22T20:45:27.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:27.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:45:27.666 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-523986010114 2026-02-22T20:45:27.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:27.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=523986010114 2026-02-22T20:45:27.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 523986010114' 2026-02-22T20:45:27.668 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 523986010114 2026-02-22T20:45:27.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:27.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 523986010116 -lt 523986010114 2026-02-22T20:45:27.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:45:27.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:27.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:28.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:45:28.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:45:28.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:45:28.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:45:28.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:45:28.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:45:28.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:45:28.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:45:28.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:45:28.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:45:28.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:28.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:28.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:45:28.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:45:28.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:45:28.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6545: TEST_dual_store_replicated_cluster: echo -n D 2026-02-22T20:45:28.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6546: TEST_dual_store_replicated_cluster: rados --pool csr_pool put ROBJ9 td/osd-scrub-repair/change 2026-02-22T20:45:28.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6547: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-22T20:45:28.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:45:28.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:45:28.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:45:28.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:45:28.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-22T20:45:28.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:45:28.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:45:28.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:45:28.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:45:28.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:45:28.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:45:28.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:45:28.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:45:28.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:45:28.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:45:28.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:45:28.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-22T20:45:28.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:45:28.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:45:28.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:45:28.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:45:28.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:28.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ9 set-attr _ td/osd-scrub-repair/robj9-oi 2026-02-22T20:45:28.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:28.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:45:28.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:45:28.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:45:28.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:45:28.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:28.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:45:28.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:45:28.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:45:28.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:45:28.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:45:28.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:45:28.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:45:28.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:45:28.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:45:28.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:45:28.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:45:28.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:45:28.996 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:45:28.997 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:45:28.997 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:45:28.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:28.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:28.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:45:28.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:45:28.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:45:28.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:45:28.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:45:28.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:45:28.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:45:28.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:45:29.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:45:29.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:45:29.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:45:29.000 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:45:29.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:29.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:45:29.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:45:29.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:45:29.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:45:29.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:45:29.018 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:29.005+0000 7fb3b6b86a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:29.019 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:29.005+0000 7fb3b6b86a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:29.021 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:29.009+0000 7fb3b6b86a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:29.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:45:29.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:45:29.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:45:29.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:45:29.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:45:29.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:29.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:45:29.160 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:45:29.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:29.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:29.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:45:29.394 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:29.381+0000 7fb3b6b86a40 -1 Falling back to public interface 2026-02-22T20:45:30.055 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:30.041+0000 7fb3b6b86a40 -1 osd.1 124 log_to_monitors true 2026-02-22T20:45:30.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:45:30.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:30.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:45:30.318 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:45:30.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:30.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:30.477 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 128 up_thru 128 down_at 125 last_clean_interval [122,124) [v2:127.0.0.1:6810/1142031601,v1:127.0.0.1:6811/1142031601] [v2:127.0.0.1:6812/1142031601,v1:127.0.0.1:6813/1142031601] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:45:30.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:45:30.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:45:30.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:45:30.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:45:30.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:45:30.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:45:30.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:45:30.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:45:30.479 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:45:30.479 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:45:30.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:45:30.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:45:30.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:45:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:45:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:45:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:45:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:45:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:45:30.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:45:30.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:45:30.759 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:45:30.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:45:30.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:30.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:45:30.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=455266533404 2026-02-22T20:45:30.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 455266533404 2026-02-22T20:45:30.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533404' 2026-02-22T20:45:30.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:30.830 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:45:30.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=549755813890 2026-02-22T20:45:30.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 549755813890 2026-02-22T20:45:30.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-455266533404 1-549755813890' 2026-02-22T20:45:30.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:30.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-455266533404 2026-02-22T20:45:30.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:30.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:45:30.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-455266533404 2026-02-22T20:45:30.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:30.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=455266533404 2026-02-22T20:45:30.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 455266533404' 2026-02-22T20:45:30.906 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 455266533404 2026-02-22T20:45:30.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:31.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533404 -lt 455266533404 2026-02-22T20:45:31.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:31.069 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-549755813890 2026-02-22T20:45:31.069 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:31.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:45:31.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-549755813890 2026-02-22T20:45:31.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:31.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=549755813890 2026-02-22T20:45:31.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 549755813890' 2026-02-22T20:45:31.073 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 549755813890 2026-02-22T20:45:31.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:31.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 549755813890 -lt 549755813890 2026-02-22T20:45:31.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:45:31.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:31.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:31.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:45:31.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:45:31.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:45:31.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:45:31.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:45:31.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:45:31.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:45:31.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:45:31.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:45:31.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:45:31.584 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:31.584 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:31.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:45:31.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:45:31.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:45:31.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6548: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/oi td/osd-scrub-repair/change 2026-02-22T20:45:31.777 INFO:tasks.workunit.client.0.trial150.stderr:rm: cannot remove 'td/osd-scrub-repair/oi': No such file or directory 2026-02-22T20:45:31.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:45:31.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ10 2026-02-22T20:45:31.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 10 % 2 2026-02-22T20:45:31.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-22T20:45:31.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:45:31.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:45:31.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ11 2026-02-22T20:45:31.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 11 % 2 2026-02-22T20:45:31.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-22T20:45:31.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:45:31.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:45:31.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ12 2026-02-22T20:45:31.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 12 % 2 2026-02-22T20:45:31.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-22T20:45:31.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:45:31.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:45:31.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ13 2026-02-22T20:45:31.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 13 % 2 2026-02-22T20:45:31.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-22T20:45:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:45:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:45:31.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ14 2026-02-22T20:45:31.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 14 % 2 2026-02-22T20:45:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-22T20:45:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:45:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6557: TEST_dual_store_replicated_cluster: echo -n bad-val 2026-02-22T20:45:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6558: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-22T20:45:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:45:31.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:45:31.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:45:31.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:45:31.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-22T20:45:31.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:45:31.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:45:31.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:45:31.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:45:31.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:45:31.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:45:31.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:45:31.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:45:31.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:45:31.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:45:32.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:45:32.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-22T20:45:32.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:45:32.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:45:32.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:45:32.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:45:32.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:45:32.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ14 set-attr _ td/osd-scrub-repair/bad-val 2026-02-22T20:45:32.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:32.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:45:32.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:45:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:45:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:45:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:45:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:45:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:45:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:45:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:45:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:45:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:45:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:45:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:45:32.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:45:32.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:45:32.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:45:32.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:45:32.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:45:32.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:45:32.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:45:32.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:45:32.372 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:45:32.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:32.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:45:32.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:45:32.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:45:32.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:45:32.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:45:32.395 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:32.381+0000 7f113e440a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:32.397 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:32.385+0000 7f113e440a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:32.398 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:32.385+0000 7f113e440a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:32.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:45:32.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:45:32.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:45:32.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:45:32.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:45:32.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:32.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:45:32.532 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:45:32.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:32.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:45:32.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:45:32.818 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:32.805+0000 7f113e440a40 -1 Falling back to public interface 2026-02-22T20:45:33.482 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:33.469+0000 7f113e440a40 -1 osd.0 130 log_to_monitors true 2026-02-22T20:45:33.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:45:33.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:33.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:45:33.690 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:45:33.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:33.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:45:33.845 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 134 up_thru 134 down_at 131 last_clean_interval [106,130) [v2:127.0.0.1:6802/1075953058,v1:127.0.0.1:6803/1075953058] [v2:127.0.0.1:6804/1075953058,v1:127.0.0.1:6805/1075953058] exists,up 976f5525-b691-48b1-b2d2-8b61723446d8 2026-02-22T20:45:33.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:45:33.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:45:33.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:45:33.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:45:33.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:45:33.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:45:33.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:45:33.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:45:33.847 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:45:33.847 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:45:33.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:45:33.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:45:33.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:45:33.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:45:33.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:45:33.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:45:33.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:45:33.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:45:33.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:45:34.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:45:34.125 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:45:34.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:45:34.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:34.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:45:34.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=575525617666 2026-02-22T20:45:34.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 575525617666 2026-02-22T20:45:34.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-575525617666' 2026-02-22T20:45:34.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:34.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:45:34.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=549755813895 2026-02-22T20:45:34.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 549755813895 2026-02-22T20:45:34.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-575525617666 1-549755813895' 2026-02-22T20:45:34.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:34.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-575525617666 2026-02-22T20:45:34.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:34.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:45:34.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-575525617666 2026-02-22T20:45:34.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:34.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=575525617666 2026-02-22T20:45:34.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 575525617666' 2026-02-22T20:45:34.274 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 575525617666 2026-02-22T20:45:34.275 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:34.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 455266533405 -lt 575525617666 2026-02-22T20:45:34.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:35.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:45:35.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:35.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617667 -lt 575525617666 2026-02-22T20:45:35.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:35.603 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-549755813895 2026-02-22T20:45:35.603 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:35.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:45:35.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-549755813895 2026-02-22T20:45:35.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:35.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=549755813895 2026-02-22T20:45:35.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 549755813895' 2026-02-22T20:45:35.607 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 549755813895 2026-02-22T20:45:35.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:35.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 549755813895 -lt 549755813895 2026-02-22T20:45:35.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:45:35.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:35.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:35.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:45:35.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:45:35.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:45:35.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:45:35.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:45:35.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:45:35.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:45:35.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:45:36.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:45:36.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:45:36.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:36.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:36.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6559: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:45:36.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:45:36.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:45:36.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:45:36.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:45:36.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:45:36.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:45:36.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:45:36.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ14 rm-attr _ 2026-02-22T20:45:36.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:45:36.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:45:36.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:45:36.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:45:36.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:36.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ14 rm-attr _ 2026-02-22T20:45:36.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:36.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:45:36.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:45:36.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:45:36.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:45:36.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:36.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:45:36.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:45:36.939 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:45:36.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:36.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:36.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:45:36.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:45:36.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:45:36.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:45:36.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:45:36.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:45:36.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:45:36.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:45:36.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:45:36.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:45:36.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:45:36.942 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:45:36.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:36.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:45:36.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:45:36.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:45:36.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:45:36.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:45:36.968 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:36.953+0000 7fc09bf28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:36.969 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:36.957+0000 7fc09bf28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:36.970 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:36.957+0000 7fc09bf28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:37.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:45:37.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:45:37.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:45:37.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:45:37.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:45:37.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:37.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:45:37.101 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:45:37.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:37.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:37.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:45:37.343 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:37.329+0000 7fc09bf28a40 -1 Falling back to public interface 2026-02-22T20:45:37.970 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:37.957+0000 7fc09bf28a40 -1 osd.1 135 log_to_monitors true 2026-02-22T20:45:38.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:45:38.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:38.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:45:38.266 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:45:38.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:38.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:38.423 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 139 up_thru 139 down_at 136 last_clean_interval [128,135) [v2:127.0.0.1:6810/1317318600,v1:127.0.0.1:6811/1317318600] [v2:127.0.0.1:6812/1317318600,v1:127.0.0.1:6813/1317318600] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:45:38.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:45:38.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:45:38.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:45:38.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:45:38.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:45:38.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:45:38.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:45:38.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:45:38.425 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:45:38.425 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:45:38.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:45:38.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:45:38.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:45:38.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:45:38.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:45:38.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:45:38.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:45:38.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:45:38.546 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:45:38.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:45:38.704 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:45:38.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:45:38.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:38.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:45:38.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=575525617672 2026-02-22T20:45:38.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 575525617672 2026-02-22T20:45:38.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-575525617672' 2026-02-22T20:45:38.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:38.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:45:38.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=597000454146 2026-02-22T20:45:38.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 597000454146 2026-02-22T20:45:38.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-575525617672 1-597000454146' 2026-02-22T20:45:38.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:38.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-575525617672 2026-02-22T20:45:38.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:38.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:45:38.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-575525617672 2026-02-22T20:45:38.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:38.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=575525617672 2026-02-22T20:45:38.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 575525617672' 2026-02-22T20:45:38.854 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 575525617672 2026-02-22T20:45:38.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:39.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617669 -lt 575525617672 2026-02-22T20:45:39.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:40.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:45:40.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:40.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617672 -lt 575525617672 2026-02-22T20:45:40.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:40.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-597000454146 2026-02-22T20:45:40.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:40.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:45:40.184 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-597000454146 2026-02-22T20:45:40.184 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:40.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=597000454146 2026-02-22T20:45:40.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 597000454146' 2026-02-22T20:45:40.186 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 597000454146 2026-02-22T20:45:40.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:40.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 597000454146 -lt 597000454146 2026-02-22T20:45:40.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:45:40.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:40.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:40.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:45:40.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:45:40.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:45:40.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:45:40.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:45:40.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:45:40.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:45:40.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:45:40.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:45:40.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:45:40.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:40.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:40.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:45:40.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:45:40.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:45:40.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6560: TEST_dual_store_replicated_cluster: rm td/osd-scrub-repair/bad-val 2026-02-22T20:45:40.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:45:40.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ15 2026-02-22T20:45:40.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 15 % 2 2026-02-22T20:45:40.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-22T20:45:40.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:45:40.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6564: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-22T20:45:40.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:45:40.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:45:40.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:45:40.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:45:40.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-22T20:45:40.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:45:40.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:45:40.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:45:40.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:45:40.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:45:40.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:45:40.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:45:40.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:45:40.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:45:40.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:45:41.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:45:41.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ15 rm-attr _ 2026-02-22T20:45:41.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:45:41.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:45:41.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:45:41.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:45:41.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:41.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ15 rm-attr _ 2026-02-22T20:45:41.384 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:45:41.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:45:41.482 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:45:41.482 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:45:41.482 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:45:41.482 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:45:41.482 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:45:41.482 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:45:41.482 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:41.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:41.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:45:41.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:45:41.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:45:41.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:45:41.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:45:41.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:45:41.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:45:41.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:45:41.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:45:41.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:45:41.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:45:41.484 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:45:41.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:41.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:45:41.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:45:41.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:45:41.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:45:41.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:45:41.502 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:41.489+0000 7f0261e90a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:41.503 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:41.489+0000 7f0261e90a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:41.504 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:41.489+0000 7f0261e90a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:41.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:45:41.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:45:41.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:45:41.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:45:41.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:45:41.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:41.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:45:41.644 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:45:41.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:41.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:41.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:45:41.842 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:41.829+0000 7f0261e90a40 -1 Falling back to public interface 2026-02-22T20:45:42.418 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:42.405+0000 7f0261e90a40 -1 osd.1 140 log_to_monitors true 2026-02-22T20:45:42.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:45:42.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:42.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:45:42.805 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:45:42.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:42.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:42.966 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 144 up_thru 144 down_at 141 last_clean_interval [139,140) [v2:127.0.0.1:6810/990041840,v1:127.0.0.1:6811/990041840] [v2:127.0.0.1:6812/990041840,v1:127.0.0.1:6813/990041840] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:45:42.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:45:42.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:45:42.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:45:42.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:45:42.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:45:42.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:45:42.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:45:42.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:45:42.968 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:45:42.968 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:45:42.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:45:42.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:45:42.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:45:43.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:45:43.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:45:43.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:45:43.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:45:43.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:45:43.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:45:43.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:45:43.241 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:45:43.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:45:43.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:43.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:45:43.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=575525617677 2026-02-22T20:45:43.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 575525617677 2026-02-22T20:45:43.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-575525617677' 2026-02-22T20:45:43.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:43.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:45:43.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=618475290626 2026-02-22T20:45:43.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 618475290626 2026-02-22T20:45:43.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-575525617677 1-618475290626' 2026-02-22T20:45:43.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:43.389 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-575525617677 2026-02-22T20:45:43.389 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:43.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:45:43.391 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-575525617677 2026-02-22T20:45:43.391 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:43.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=575525617677 2026-02-22T20:45:43.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 575525617677' 2026-02-22T20:45:43.393 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 575525617677 2026-02-22T20:45:43.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:43.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617676 -lt 575525617677 2026-02-22T20:45:43.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:44.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:45:44.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:44.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617676 -lt 575525617677 2026-02-22T20:45:44.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:45.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:45:45.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:45.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 575525617679 -lt 575525617677 2026-02-22T20:45:45.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:45.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-618475290626 2026-02-22T20:45:45.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:45.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:45:45.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-618475290626 2026-02-22T20:45:45.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:45.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=618475290626 2026-02-22T20:45:45.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 618475290626' 2026-02-22T20:45:45.876 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 618475290626 2026-02-22T20:45:45.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:46.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 618475290628 -lt 618475290626 2026-02-22T20:45:46.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:45:46.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:46.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:46.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:45:46.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:45:46.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:45:46.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:45:46.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:45:46.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:45:46.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:45:46.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:45:46.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:45:46.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:45:46.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:46.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:46.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:45:46.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:45:46.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:45:46.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:45:46.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ16 2026-02-22T20:45:46.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 16 % 2 2026-02-22T20:45:46.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-22T20:45:46.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:45:46.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6568: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-22T20:45:46.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:45:46.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:45:46.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:45:46.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:45:46.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-22T20:45:46.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:45:46.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:45:46.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:45:46.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:45:46.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:45:46.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:45:46.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:45:46.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:45:46.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:45:46.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:45:46.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:45:46.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ16 rm-attr snapset 2026-02-22T20:45:46.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:45:46.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:45:46.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:45:46.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:45:46.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:45:46.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ16 rm-attr snapset 2026-02-22T20:45:47.093 INFO:tasks.workunit.client.0.trial150.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-22T20:45:47.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:47.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:45:47.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:45:47.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:45:47.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:45:47.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:45:47.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:45:47.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:45:47.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:45:47.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:45:47.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:45:47.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:45:47.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:45:47.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:45:47.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:45:47.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:45:47.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:45:47.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:45:47.208 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:45:47.208 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:45:47.208 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:45:47.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:45:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:45:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:45:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:45:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:45:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:45:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:45:47.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:45:47.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:45:47.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:45:47.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:45:47.211 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:45:47.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:47.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:45:47.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:45:47.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:45:47.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:45:47.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:45:47.230 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:47.217+0000 7f5ba9116a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:47.231 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:47.217+0000 7f5ba9116a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:47.232 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:47.217+0000 7f5ba9116a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:47.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:45:47.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:45:47.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:45:47.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:45:47.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:45:47.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:47.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:45:47.371 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:45:47.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:47.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:45:47.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:45:47.674 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:47.661+0000 7f5ba9116a40 -1 Falling back to public interface 2026-02-22T20:45:48.263 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:48.249+0000 7f5ba9116a40 -1 osd.0 145 log_to_monitors true 2026-02-22T20:45:48.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:45:48.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:48.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:45:48.526 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:45:48.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:48.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:45:48.685 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 149 up_thru 149 down_at 146 last_clean_interval [134,145) [v2:127.0.0.1:6802/3355771767,v1:127.0.0.1:6803/3355771767] [v2:127.0.0.1:6804/3355771767,v1:127.0.0.1:6805/3355771767] exists,up 976f5525-b691-48b1-b2d2-8b61723446d8 2026-02-22T20:45:48.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:45:48.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:45:48.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:45:48.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:45:48.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:45:48.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:45:48.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:45:48.686 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:45:48.687 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:45:48.687 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:45:48.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:45:48.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:45:48.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:45:48.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:45:48.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:45:48.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:45:48.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:45:48.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:45:48.808 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:45:48.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:45:48.970 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:45:48.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:45:48.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:48.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:45:49.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=639950127106 2026-02-22T20:45:49.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 639950127106 2026-02-22T20:45:49.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127106' 2026-02-22T20:45:49.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:49.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:45:49.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=618475290633 2026-02-22T20:45:49.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 618475290633 2026-02-22T20:45:49.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127106 1-618475290633' 2026-02-22T20:45:49.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:49.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-639950127106 2026-02-22T20:45:49.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:49.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:45:49.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-639950127106 2026-02-22T20:45:49.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:49.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=639950127106 2026-02-22T20:45:49.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 639950127106' 2026-02-22T20:45:49.120 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 639950127106 2026-02-22T20:45:49.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:49.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 639950127106 2026-02-22T20:45:49.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:50.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:45:50.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:50.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 639950127106 2026-02-22T20:45:50.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:51.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:45:51.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:51.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127108 -lt 639950127106 2026-02-22T20:45:51.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:51.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-618475290633 2026-02-22T20:45:51.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:51.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:45:51.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-618475290633 2026-02-22T20:45:51.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:51.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=618475290633 2026-02-22T20:45:51.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 618475290633' 2026-02-22T20:45:51.603 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 618475290633 2026-02-22T20:45:51.603 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:51.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 618475290635 -lt 618475290633 2026-02-22T20:45:51.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:45:51.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:51.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:51.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:45:51.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:45:51.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:45:51.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:45:51.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:45:51.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:45:51.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:45:51.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:45:52.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:45:52.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:45:52.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:52.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6569: TEST_dual_store_replicated_cluster: echo -n bad-val 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6570: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:45:52.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:45:52.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:45:52.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:45:52.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:45:52.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:45:52.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:45:52.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:45:52.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-22T20:45:52.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:45:52.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:45:52.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:45:52.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:45:52.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:52.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ16 set-attr snapset td/osd-scrub-repair/bad-val 2026-02-22T20:45:52.817 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-22T20:45:52.817 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:45:52.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:45:52.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:45:52.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:45:52.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:45:52.906 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:45:52.906 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:45:52.906 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:45:52.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:52.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:52.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:45:52.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:45:52.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:45:52.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:45:52.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:45:52.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:45:52.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:45:52.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:45:52.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:45:52.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:45:52.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:45:52.908 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:45:52.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:52.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:45:52.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:45:52.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:45:52.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:45:52.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:45:52.929 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:52.917+0000 7f17382dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:52.931 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:52.917+0000 7f17382dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:52.932 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:52.917+0000 7f17382dca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:53.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:45:53.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:45:53.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:45:53.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:45:53.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:45:53.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:53.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:45:53.070 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:45:53.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:53.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:53.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:45:53.362 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:53.349+0000 7f17382dca40 -1 Falling back to public interface 2026-02-22T20:45:54.005 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:53.993+0000 7f17382dca40 -1 osd.1 150 log_to_monitors true 2026-02-22T20:45:54.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:45:54.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:54.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:45:54.232 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:45:54.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:54.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:45:54.392 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 154 up_thru 154 down_at 151 last_clean_interval [144,150) [v2:127.0.0.1:6810/2453919672,v1:127.0.0.1:6811/2453919672] [v2:127.0.0.1:6812/2453919672,v1:127.0.0.1:6813/2453919672] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:45:54.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:45:54.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:45:54.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:45:54.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:45:54.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:45:54.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:45:54.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:45:54.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:45:54.394 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:45:54.394 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:45:54.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:45:54.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:45:54.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:45:54.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:45:54.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:45:54.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:45:54.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:45:54.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:45:54.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:45:54.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:45:54.678 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:45:54.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:45:54.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:54.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:45:54.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=639950127113 2026-02-22T20:45:54.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 639950127113 2026-02-22T20:45:54.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127113' 2026-02-22T20:45:54.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:54.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:45:54.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=661424963586 2026-02-22T20:45:54.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 661424963586 2026-02-22T20:45:54.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-639950127113 1-661424963586' 2026-02-22T20:45:54.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:54.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-639950127113 2026-02-22T20:45:54.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:54.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:45:54.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-639950127113 2026-02-22T20:45:54.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:54.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=639950127113 2026-02-22T20:45:54.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 639950127113' 2026-02-22T20:45:54.829 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 639950127113 2026-02-22T20:45:54.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:54.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127110 -lt 639950127113 2026-02-22T20:45:54.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:45:55.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:45:55.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:56.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 639950127113 -lt 639950127113 2026-02-22T20:45:56.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:56.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-661424963586 2026-02-22T20:45:56.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:56.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:45:56.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-661424963586 2026-02-22T20:45:56.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:56.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=661424963586 2026-02-22T20:45:56.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 661424963586' 2026-02-22T20:45:56.161 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 661424963586 2026-02-22T20:45:56.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:45:56.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 661424963586 -lt 661424963586 2026-02-22T20:45:56.327 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:45:56.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:56.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:56.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:45:56.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:45:56.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:45:56.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:45:56.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:45:56.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:45:56.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:45:56.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:45:56.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:45:56.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:45:56.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:45:56.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:45:56.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:45:56.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:45:56.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:45:56.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:45:56.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ17 2026-02-22T20:45:56.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 17 % 2 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6575: TEST_dual_store_replicated_cluster: local payload=ROBJ17 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6576: TEST_dual_store_replicated_cluster: echo ROBJ17 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6577: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:45:56.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:45:56.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:45:56.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:45:56.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:45:56.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:45:56.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:45:56.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:45:56.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:45:57.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:45:57.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:45:57.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:45:57.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:45:57.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:45:57.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:45:57.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:45:57.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:45:57.381 INFO:tasks.workunit.client.0.trial150.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-22T20:45:57.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:57.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:45:57.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:45:57.485 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:45:57.486 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:45:57.486 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:45:57.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:57.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:45:57.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:45:57.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:45:57.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:45:57.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:45:57.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:45:57.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:45:57.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:45:57.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:45:57.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:45:57.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:45:57.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:45:57.488 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:45:57.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:45:57.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:45:57.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:45:57.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:45:57.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:45:57.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:45:57.506 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:57.494+0000 7fe75929ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:57.508 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:57.498+0000 7fe75929ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:57.509 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:57.498+0000 7fe75929ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:45:57.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:45:57.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:45:57.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:45:57.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:45:57.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:45:57.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:57.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:45:57.651 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:45:57.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:57.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:45:57.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:45:57.886 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:57.874+0000 7fe75929ba40 -1 Falling back to public interface 2026-02-22T20:45:58.478 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:45:58.466+0000 7fe75929ba40 -1 osd.0 156 log_to_monitors true 2026-02-22T20:45:58.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:45:58.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:45:58.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:45:58.811 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:45:58.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:45:58.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:45:58.964 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 160 up_thru 160 down_at 157 last_clean_interval [149,156) [v2:127.0.0.1:6802/1935155490,v1:127.0.0.1:6803/1935155490] [v2:127.0.0.1:6804/1935155490,v1:127.0.0.1:6805/1935155490] exists,up 976f5525-b691-48b1-b2d2-8b61723446d8 2026-02-22T20:45:58.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:45:58.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:45:58.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:45:58.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:45:58.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:45:58.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:45:58.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:45:58.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:45:58.966 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:45:58.966 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:45:58.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:45:58.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:45:58.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:45:59.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:45:59.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:45:59.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:45:59.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:45:59.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:45:59.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:45:59.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:45:59.242 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:45:59.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:45:59.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:59.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:45:59.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=687194767362 2026-02-22T20:45:59.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 687194767362 2026-02-22T20:45:59.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-687194767362' 2026-02-22T20:45:59.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:45:59.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:45:59.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=661424963592 2026-02-22T20:45:59.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 661424963592 2026-02-22T20:45:59.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-687194767362 1-661424963592' 2026-02-22T20:45:59.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:45:59.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-687194767362 2026-02-22T20:45:59.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:45:59.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:45:59.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-687194767362 2026-02-22T20:45:59.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:45:59.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=687194767362 2026-02-22T20:45:59.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 687194767362' 2026-02-22T20:45:59.394 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 687194767362 2026-02-22T20:45:59.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:45:59.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 687194767362 2026-02-22T20:45:59.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:00.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:46:00.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:00.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 687194767362 2026-02-22T20:46:00.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:01.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:46:01.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:01.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 687194767364 -lt 687194767362 2026-02-22T20:46:01.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:01.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-661424963592 2026-02-22T20:46:01.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:01.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:46:01.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-661424963592 2026-02-22T20:46:01.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:01.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=661424963592 2026-02-22T20:46:01.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 661424963592' 2026-02-22T20:46:01.896 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 661424963592 2026-02-22T20:46:01.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:02.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 661424963593 -lt 661424963592 2026-02-22T20:46:02.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:46:02.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:46:02.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:46:02.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:46:02.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:46:02.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:46:02.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:46:02.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:46:02.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:46:02.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:46:02.254 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:46:02.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:46:02.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:46:02.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:46:02.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:46:02.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:46:02.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:46:02.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:46:02.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6578: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:46:02.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:46:02.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:46:02.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:46:02.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:46:02.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:46:02.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:46:02.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:46:02.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:46:02.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:46:02.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:46:02.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:46:02.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:46:02.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:46:02.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:46:02.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:46:02.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:46:02.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:46:02.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:46:02.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:46:02.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:46:02.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:46:02.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:46:02.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ17 set-bytes td/osd-scrub-repair/new.ROBJ17 2026-02-22T20:46:03.111 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-22T20:46:03.111 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-22T20:46:03.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:46:03.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:46:03.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:46:03.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:46:03.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:46:03.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:46:03.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:46:03.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:46:03.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:46:03.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:46:03.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:46:03.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:46:03.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:46:03.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:46:03.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:46:03.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:46:03.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:46:03.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:46:03.204 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:46:03.204 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:03.204 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:03.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:03.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:03.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:46:03.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:46:03.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:46:03.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:46:03.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:46:03.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:46:03.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:46:03.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:46:03.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:46:03.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:46:03.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:46:03.207 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:46:03.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:46:03.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:46:03.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:46:03.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:46:03.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:46:03.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:46:03.225 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:03.214+0000 7f2c0e92ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:03.227 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:03.214+0000 7f2c0e92ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:03.228 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:03.218+0000 7f2c0e92ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:03.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:46:03.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:46:03.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:46:03.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:46:03.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:46:03.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:03.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:46:03.360 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:46:03.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:03.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:46:03.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:46:03.590 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:03.578+0000 7f2c0e92ba40 -1 Falling back to public interface 2026-02-22T20:46:04.162 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:04.150+0000 7f2c0e92ba40 -1 osd.1 161 log_to_monitors true 2026-02-22T20:46:04.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:46:04.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:04.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:46:04.522 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:46:04.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:04.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:46:04.680 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 165 up_thru 165 down_at 162 last_clean_interval [154,161) [v2:127.0.0.1:6810/209809300,v1:127.0.0.1:6811/209809300] [v2:127.0.0.1:6812/209809300,v1:127.0.0.1:6813/209809300] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:46:04.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:46:04.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:46:04.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:46:04.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:46:04.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:46:04.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:46:04.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:46:04.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:46:04.682 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:46:04.682 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:46:04.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:46:04.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:46:04.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:46:04.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:46:04.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:46:04.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:46:04.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:46:04.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:46:04.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:46:04.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:46:04.966 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:46:04.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:46:04.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:04.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:46:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=687194767369 2026-02-22T20:46:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 687194767369 2026-02-22T20:46:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-687194767369' 2026-02-22T20:46:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:05.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:46:05.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=708669603842 2026-02-22T20:46:05.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 708669603842 2026-02-22T20:46:05.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-687194767369 1-708669603842' 2026-02-22T20:46:05.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:05.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-687194767369 2026-02-22T20:46:05.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:05.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:46:05.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-687194767369 2026-02-22T20:46:05.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:05.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=687194767369 2026-02-22T20:46:05.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 687194767369' 2026-02-22T20:46:05.116 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 687194767369 2026-02-22T20:46:05.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:05.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 687194767368 -lt 687194767369 2026-02-22T20:46:05.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:06.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:46:06.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:06.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 687194767368 -lt 687194767369 2026-02-22T20:46:06.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:07.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:46:07.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:07.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 687194767371 -lt 687194767369 2026-02-22T20:46:07.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:07.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-708669603842 2026-02-22T20:46:07.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:07.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:46:07.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-708669603842 2026-02-22T20:46:07.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:07.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=708669603842 2026-02-22T20:46:07.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 708669603842' 2026-02-22T20:46:07.611 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 708669603842 2026-02-22T20:46:07.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:07.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 708669603844 -lt 708669603842 2026-02-22T20:46:07.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:46:07.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:46:07.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:46:07.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:46:07.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:46:07.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:46:07.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:46:07.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:46:07.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:46:07.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:46:07.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:46:08.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:46:08.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:46:08.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:46:08.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:46:08.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:46:08.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:46:08.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:46:08.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:46:08.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ18 2026-02-22T20:46:08.327 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 18 % 2 2026-02-22T20:46:08.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=0 2026-02-22T20:46:08.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:46:08.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6583: TEST_dual_store_replicated_cluster: local payload=ROBJ18 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6584: TEST_dual_store_replicated_cluster: echo ROBJ18 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6585: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:46:08.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:46:08.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:46:08.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:46:08.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:46:08.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:46:08.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:46:08.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:46:08.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:46:08.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:46:08.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:46:08.800 INFO:tasks.workunit.client.0.trial150.stderr:Error decoding attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (22) Invalid argument 2026-02-22T20:46:08.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:46:08.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:46:08.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:46:08.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:46:08.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:46:08.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:46:08.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:46:08.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:46:08.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:46:08.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:46:08.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:46:08.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:46:08.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:46:08.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:46:08.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:46:08.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:46:08.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:46:08.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:46:08.900 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:46:08.900 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:08.900 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:08.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:46:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:46:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:46:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:46:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:46:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:46:08.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:46:08.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:46:08.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:46:08.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:46:08.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:46:08.903 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:46:08.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:46:08.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:46:08.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:46:08.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:46:08.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:46:08.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:46:08.921 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:08.910+0000 7f8f27412a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:08.923 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:08.910+0000 7f8f27412a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:08.924 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:08.914+0000 7f8f27412a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:09.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:46:09.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:46:09.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:46:09.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:46:09.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:46:09.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:09.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:46:09.061 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:46:09.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:09.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:46:09.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:46:09.326 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:09.314+0000 7f8f27412a40 -1 Falling back to public interface 2026-02-22T20:46:09.979 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:09.966+0000 7f8f27412a40 -1 osd.0 167 log_to_monitors true 2026-02-22T20:46:10.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:46:10.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:10.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:46:10.218 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:46:10.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:10.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:46:10.380 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 171 up_thru 171 down_at 168 last_clean_interval [160,167) [v2:127.0.0.1:6802/3240787707,v1:127.0.0.1:6803/3240787707] [v2:127.0.0.1:6804/3240787707,v1:127.0.0.1:6805/3240787707] exists,up 976f5525-b691-48b1-b2d2-8b61723446d8 2026-02-22T20:46:10.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:46:10.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:46:10.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:46:10.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:46:10.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:46:10.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:46:10.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:46:10.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:46:10.382 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:46:10.382 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:46:10.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:46:10.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:46:10.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:46:10.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:46:10.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:46:10.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:46:10.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:46:10.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:46:10.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:46:10.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:46:10.658 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:46:10.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:46:10.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:10.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:46:10.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407618 2026-02-22T20:46:10.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407618 2026-02-22T20:46:10.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407618' 2026-02-22T20:46:10.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:10.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:46:10.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=708669603849 2026-02-22T20:46:10.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 708669603849 2026-02-22T20:46:10.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407618 1-708669603849' 2026-02-22T20:46:10.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:10.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407618 2026-02-22T20:46:10.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:10.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:46:10.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407618 2026-02-22T20:46:10.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:10.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407618 2026-02-22T20:46:10.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407618' 2026-02-22T20:46:10.808 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 734439407618 2026-02-22T20:46:10.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:10.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 687194767372 -lt 734439407618 2026-02-22T20:46:10.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:11.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:46:11.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:12.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407618 -lt 734439407618 2026-02-22T20:46:12.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:12.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-708669603849 2026-02-22T20:46:12.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:12.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:46:12.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-708669603849 2026-02-22T20:46:12.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:12.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=708669603849 2026-02-22T20:46:12.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 708669603849' 2026-02-22T20:46:12.130 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 708669603849 2026-02-22T20:46:12.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:12.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 708669603849 -lt 708669603849 2026-02-22T20:46:12.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:46:12.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:46:12.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:46:12.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:46:12.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:46:12.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:46:12.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:46:12.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:46:12.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:46:12.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:46:12.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:46:12.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:46:12.642 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:46:12.642 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:46:12.642 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:46:12.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:46:12.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:46:12.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:46:12.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6586: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:46:12.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:46:12.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:46:12.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:46:12.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:46:12.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:46:12.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:46:12.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:46:12.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:46:12.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:46:12.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:46:12.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:46:12.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:46:12.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:46:12.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:46:12.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:46:13.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:46:13.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:46:13.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:46:13.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:46:13.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:46:13.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:46:13.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:46:13.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 set-bytes td/osd-scrub-repair/new.ROBJ18 2026-02-22T20:46:13.305 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-22T20:46:13.305 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-22T20:46:13.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:46:13.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:46:13.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:46:13.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:46:13.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:46:13.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:46:13.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:46:13.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:46:13.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:46:13.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:46:13.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:46:13.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:46:13.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:46:13.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:46:13.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:46:13.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:46:13.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:46:13.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:46:13.407 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:46:13.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:46:13.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:46:13.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:46:13.410 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:46:13.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:46:13.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:46:13.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:46:13.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:46:13.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:46:13.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:46:13.426 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:13.414+0000 7ffb6f5f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:13.428 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:13.418+0000 7ffb6f5f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:13.429 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:13.418+0000 7ffb6f5f4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:13.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:46:13.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:46:13.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:46:13.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:46:13.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:46:13.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:13.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:46:13.569 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:46:13.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:13.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:46:13.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:46:13.802 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:13.790+0000 7ffb6f5f4a40 -1 Falling back to public interface 2026-02-22T20:46:14.383 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:14.370+0000 7ffb6f5f4a40 -1 osd.1 172 log_to_monitors true 2026-02-22T20:46:14.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:46:14.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:14.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:46:14.726 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:46:14.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:14.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:46:14.882 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 176 up_thru 176 down_at 173 last_clean_interval [165,172) [v2:127.0.0.1:6810/3632462240,v1:127.0.0.1:6811/3632462240] [v2:127.0.0.1:6812/3632462240,v1:127.0.0.1:6813/3632462240] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:46:14.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:46:14.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:46:14.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:46:14.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:46:14.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:46:14.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:46:14.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:46:14.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:46:14.884 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:46:14.884 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:46:14.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:46:14.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:46:14.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:46:14.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:46:14.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:46:14.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:46:14.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:46:14.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:46:14.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:46:15.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:46:15.158 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:46:15.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:46:15.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:15.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:46:15.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407624 2026-02-22T20:46:15.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407624 2026-02-22T20:46:15.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407624' 2026-02-22T20:46:15.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:15.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:46:15.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=755914244098 2026-02-22T20:46:15.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 755914244098 2026-02-22T20:46:15.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407624 1-755914244098' 2026-02-22T20:46:15.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:15.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407624 2026-02-22T20:46:15.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:15.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:46:15.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407624 2026-02-22T20:46:15.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:15.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407624 2026-02-22T20:46:15.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407624' 2026-02-22T20:46:15.308 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 734439407624 2026-02-22T20:46:15.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:15.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407622 -lt 734439407624 2026-02-22T20:46:15.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:16.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:46:16.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:16.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407622 -lt 734439407624 2026-02-22T20:46:16.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:17.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:46:17.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:17.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407625 -lt 734439407624 2026-02-22T20:46:17.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:17.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-755914244098 2026-02-22T20:46:17.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:17.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:46:17.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-755914244098 2026-02-22T20:46:17.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:17.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=755914244098 2026-02-22T20:46:17.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 755914244098' 2026-02-22T20:46:17.803 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 755914244098 2026-02-22T20:46:17.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:17.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 755914244100 -lt 755914244098 2026-02-22T20:46:17.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:46:17.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:46:17.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:46:18.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:46:18.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:46:18.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:46:18.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:46:18.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:46:18.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:46:18.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:46:18.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:46:18.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:46:18.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:46:18.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:46:18.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:46:18.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:46:18.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:46:18.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:46:18.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6588: TEST_dual_store_replicated_cluster: objectstore_tool td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-22T20:46:18.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:46:18.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:46:18.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T20:46:18.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:46:18.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-22T20:46:18.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:46:18.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:46:18.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T20:46:18.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:46:18.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:46:18.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:46:18.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:46:18.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:46:18.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:46:18.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:46:18.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:46:18.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 1 ROBJ18 corrupt-info 2026-02-22T20:46:18.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:46:18.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:46:18.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:46:18.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:46:18.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:46:18.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 ROBJ18 corrupt-info 2026-02-22T20:46:19.019 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:30259878:::ROBJ15:head#, (61) No data available 2026-02-22T20:46:19.020 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 3.0_head,#3:c0c86b1d:::ROBJ14:head#, (61) No data available 2026-02-22T20:46:19.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:46:19.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:46:19.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:46:19.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:46:19.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:46:19.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:46:19.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:46:19.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:46:19.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:46:19.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:46:19.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:46:19.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:46:19.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:46:19.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:46:19.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:46:19.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:46:19.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:46:19.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:46:19.122 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:46:19.122 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:19.122 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:19.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:19.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:19.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:46:19.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:46:19.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:46:19.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:46:19.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:46:19.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:46:19.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:46:19.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:46:19.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0' 2026-02-22T20:46:19.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:46:19.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:46:19.126 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:46:19.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --osd_op_queue=wpq --osd_scrub_auto_repair=0 2026-02-22T20:46:19.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:46:19.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:46:19.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:46:19.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:46:19.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:46:19.147 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:19.134+0000 7fc2f21bca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:19.149 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:19.138+0000 7fc2f21bca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:19.149 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:19.138+0000 7fc2f21bca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:46:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:46:19.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:46:19.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:46:19.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:46:19.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:19.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:46:19.285 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:46:19.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:19.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:46:19.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:46:19.557 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:19.546+0000 7fc2f21bca40 -1 Falling back to public interface 2026-02-22T20:46:20.112 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:20.098+0000 7fc2f21bca40 -1 osd.1 177 log_to_monitors true 2026-02-22T20:46:20.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:46:20.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:20.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:46:20.443 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:46:20.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:20.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:46:20.597 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 181 up_thru 181 down_at 178 last_clean_interval [176,177) [v2:127.0.0.1:6810/3111594665,v1:127.0.0.1:6811/3111594665] [v2:127.0.0.1:6812/3111594665,v1:127.0.0.1:6813/3111594665] exists,up 6f50b59b-4813-480d-aedb-da74f6769de9 2026-02-22T20:46:20.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:46:20.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:46:20.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:46:20.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:46:20.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:46:20.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:46:20.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:46:20.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:46:20.598 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:46:20.598 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:46:20.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:46:20.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:46:20.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:46:20.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:46:20.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:46:20.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:46:20.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:46:20.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:46:20.714 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:46:20.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:46:20.876 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:46:20.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:46:20.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:20.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:46:20.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407630 2026-02-22T20:46:20.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407630 2026-02-22T20:46:20.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407630' 2026-02-22T20:46:20.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:20.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:46:21.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080578 2026-02-22T20:46:21.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080578 2026-02-22T20:46:21.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407630 1-777389080578' 2026-02-22T20:46:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:21.023 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407630 2026-02-22T20:46:21.023 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:21.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:46:21.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407630 2026-02-22T20:46:21.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:21.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407630 2026-02-22T20:46:21.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407630' 2026-02-22T20:46:21.027 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 734439407630 2026-02-22T20:46:21.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:21.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407630 -lt 734439407630 2026-02-22T20:46:21.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:21.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080578 2026-02-22T20:46:21.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:21.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:46:21.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080578 2026-02-22T20:46:21.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:21.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080578 2026-02-22T20:46:21.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080578' 2026-02-22T20:46:21.203 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 777389080578 2026-02-22T20:46:21.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:21.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 777389080578 2026-02-22T20:46:21.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:22.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:46:22.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:22.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 777389080578 2026-02-22T20:46:22.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:23.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:46:23.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:23.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080580 -lt 777389080578 2026-02-22T20:46:23.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:46:23.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:46:23.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:46:23.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:46:23.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:46:23.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:46:23.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:46:23.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:46:23.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:46:23.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:46:23.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:46:24.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:46:24.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:46:24.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:46:24.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:46:24.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:46:24.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:46:24.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:46:24.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6480: TEST_dual_store_replicated_cluster: for i in $(seq 1 $total_objs) 2026-02-22T20:46:24.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6481: TEST_dual_store_replicated_cluster: objname=ROBJ19 2026-02-22T20:46:24.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: expr 19 % 2 2026-02-22T20:46:24.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6484: TEST_dual_store_replicated_cluster: local osd=1 2026-02-22T20:46:24.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6486: TEST_dual_store_replicated_cluster: case $i in 2026-02-22T20:46:24.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6597: TEST_dual_store_replicated_cluster: get_pg csr_pool ROBJ0 2026-02-22T20:46:24.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=csr_pool 2026-02-22T20:46:24.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=ROBJ0 2026-02-22T20:46:24.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map csr_pool ROBJ0 2026-02-22T20:46:24.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:46:24.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6597: TEST_dual_store_replicated_cluster: local pg=3.0 2026-02-22T20:46:24.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6599: TEST_dual_store_replicated_cluster: ceph tell 'osd.*' injectargs -- --osd-max-object-size=1048576 2026-02-22T20:46:24.472 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: {} 2026-02-22T20:46:24.472 INFO:tasks.workunit.client.0.trial150.stderr:osd.0: osd_max_object_size = '' 2026-02-22T20:46:24.481 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: {} 2026-02-22T20:46:24.481 INFO:tasks.workunit.client.0.trial150.stderr:osd.1: osd_max_object_size = '' 2026-02-22T20:46:24.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6601: TEST_dual_store_replicated_cluster: inject_eio rep data csr_pool ROBJ11 td/osd-scrub-repair 0 2026-02-22T20:46:24.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-22T20:46:24.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-22T20:46:24.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-22T20:46:24.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-22T20:46:24.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-22T20:46:24.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-22T20:46:24.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ11 2026-02-22T20:46:24.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-22T20:46:24.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-22T20:46:24.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-22T20:46:24.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-22T20:46:24.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-22T20:46:24.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ11 2026-02-22T20:46:24.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-22T20:46:24.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ11 2026-02-22T20:46:24.493 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ11 2026-02-22T20:46:24.493 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:46:24.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:46:24.658 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:46:24.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-22T20:46:24.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-22T20:46:24.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-22T20:46:24.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-22T20:46:24.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-22T20:46:24.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-22T20:46:24.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-22T20:46:24.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-22T20:46:24.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-22T20:46:24.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:46:24.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-22T20:46:24.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T20:46:24.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:46:24.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:46:24.663 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-22T20:46:24.663 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:46:24.663 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:46:24.663 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:46:24.663 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:24.663 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:24.664 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:46:24.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-22T20:46:24.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:46:24.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-22T20:46:24.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-22T20:46:24.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-22T20:46:24.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:46:24.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:46:24.722 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:46:24.722 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:24.722 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:24.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:46:24.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-22T20:46:24.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.40073/ceph-osd.1.asok injectdataerr csr_pool ROBJ11 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6602: TEST_dual_store_replicated_cluster: inject_eio rep mdata csr_pool ROBJ12 td/osd-scrub-repair 1 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ12 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-22T20:46:24.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-22T20:46:24.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ12 2026-02-22T20:46:24.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-22T20:46:24.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ12 2026-02-22T20:46:24.783 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ12 2026-02-22T20:46:24.783 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:46:24.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:46:24.945 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:46:24.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-22T20:46:24.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-22T20:46:24.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-22T20:46:24.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-22T20:46:24.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-22T20:46:24.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-22T20:46:24.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-22T20:46:24.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-22T20:46:24.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-22T20:46:24.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:46:24.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-22T20:46:24.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T20:46:24.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:46:24.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:46:24.949 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-22T20:46:24.949 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:46:24.949 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:46:24.949 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:46:24.949 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:24.949 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:24.950 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:46:24.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-22T20:46:25.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:46:25.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-22T20:46:25.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-22T20:46:25.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-22T20:46:25.013 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:46:25.013 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:46:25.013 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:46:25.013 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:25.014 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:25.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:46:25.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-22T20:46:25.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.40073/ceph-osd.0.asok injectmdataerr csr_pool ROBJ12 2026-02-22T20:46:25.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6603: TEST_dual_store_replicated_cluster: inject_eio rep data csr_pool ROBJ13 td/osd-scrub-repair 0 2026-02-22T20:46:25.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-22T20:46:25.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-22T20:46:25.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=data 2026-02-22T20:46:25.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-22T20:46:25.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-22T20:46:25.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-22T20:46:25.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-22T20:46:25.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-22T20:46:25.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-22T20:46:25.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-22T20:46:25.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=0 2026-02-22T20:46:25.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-22T20:46:25.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-22T20:46:25.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-22T20:46:25.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-22T20:46:25.075 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-22T20:46:25.075 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:46:25.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:46:25.241 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:46:25.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-22T20:46:25.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-22T20:46:25.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-22T20:46:25.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=1 2026-02-22T20:46:25.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-22T20:46:25.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-22T20:46:25.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/1/type 2026-02-22T20:46:25.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-22T20:46:25.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2026-02-22T20:46:25.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:46:25.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2026-02-22T20:46:25.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T20:46:25.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:46:25.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:46:25.245 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2026-02-22T20:46:25.245 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:46:25.245 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:46:25.245 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:46:25.245 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:25.245 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:25.245 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:46:25.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2026-02-22T20:46:25.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:46:25.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-22T20:46:25.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-22T20:46:25.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.1 2026-02-22T20:46:25.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:46:25.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:46:25.307 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:46:25.307 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:25.307 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:25.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.1.asok 2026-02-22T20:46:25.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-22T20:46:25.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.40073/ceph-osd.1.asok injectdataerr csr_pool ROBJ13 2026-02-22T20:46:25.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6606: TEST_dual_store_replicated_cluster: pg_scrub 3.0 2026-02-22T20:46:25.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-22T20:46:25.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-22T20:46:25.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:46:25.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:46:25.366 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:46:25.366 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:46:25.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:46:25.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:46:25.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:46:25.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:46:25.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:46:25.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:46:25.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:46:25.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:46:25.680 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:46:25.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:46:25.835 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:46:25.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:46:25.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:25.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:46:25.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407636 2026-02-22T20:46:25.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407636 2026-02-22T20:46:25.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407636' 2026-02-22T20:46:25.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:25.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:46:25.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080584 2026-02-22T20:46:25.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080584 2026-02-22T20:46:25.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407636 1-777389080584' 2026-02-22T20:46:25.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:25.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407636 2026-02-22T20:46:25.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:25.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:46:25.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407636 2026-02-22T20:46:25.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:25.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407636 2026-02-22T20:46:25.983 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 734439407636 2026-02-22T20:46:25.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407636' 2026-02-22T20:46:25.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:26.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407634 -lt 734439407636 2026-02-22T20:46:26.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:27.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:46:27.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:27.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407637 -lt 734439407636 2026-02-22T20:46:27.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:27.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080584 2026-02-22T20:46:27.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:27.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:46:27.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080584 2026-02-22T20:46:27.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:27.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080584 2026-02-22T20:46:27.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080584' 2026-02-22T20:46:27.317 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 777389080584 2026-02-22T20:46:27.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:27.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080585 -lt 777389080584 2026-02-22T20:46:27.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:46:27.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:46:27.477 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:46:27.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:46:27.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:46:27.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:46:27.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:46:27.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:46:27.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T20:46:27.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:46:27.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:46:27.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:46:27.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-22T20:46:27.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:27.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:46:27.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:27.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:46:27.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-22T20:44:27.072684+0000 2026-02-22T20:46:27.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-22T20:46:27.853 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-22T20:46:27.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-22T20:44:27.072684+0000 2026-02-22T20:46:27.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:46:27.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:44:27.072684+0000 2026-02-22T20:46:27.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:46:27.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:46:27.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:27.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:46:27.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:27.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:46:27.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:27.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:46:28.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:44:27.072684+0000 '>' 2026-02-22T20:44:27.072684+0000 2026-02-22T20:46:28.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:46:29.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:46:29.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:29.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:46:29.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:29.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:46:29.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:29.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:46:29.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:46:28.066653+0000 '>' 2026-02-22T20:44:27.072684+0000 2026-02-22T20:46:29.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:46:29.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6607: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:29.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6608: TEST_dual_store_replicated_cluster: pg_scrub 3.0 2026-02-22T20:46:29.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-22T20:46:29.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-22T20:46:29.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:46:29.184 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:46:29.185 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:46:29.185 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:46:29.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:46:29.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:46:29.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:46:29.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:46:29.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:46:29.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:46:29.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:46:29.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:46:29.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:46:29.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:46:29.645 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:46:29.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:46:29.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:29.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:46:29.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407641 2026-02-22T20:46:29.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407641 2026-02-22T20:46:29.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407641' 2026-02-22T20:46:29.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:29.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:46:29.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080589 2026-02-22T20:46:29.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080589 2026-02-22T20:46:29.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407641 1-777389080589' 2026-02-22T20:46:29.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:29.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407641 2026-02-22T20:46:29.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:29.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:46:29.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407641 2026-02-22T20:46:29.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:29.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407641 2026-02-22T20:46:29.798 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 734439407641 2026-02-22T20:46:29.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407641' 2026-02-22T20:46:29.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:29.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407639 -lt 734439407641 2026-02-22T20:46:29.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:30.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:46:30.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:31.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407642 -lt 734439407641 2026-02-22T20:46:31.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:31.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080589 2026-02-22T20:46:31.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:31.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:46:31.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080589 2026-02-22T20:46:31.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:31.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080589 2026-02-22T20:46:31.139 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 777389080589 2026-02-22T20:46:31.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080589' 2026-02-22T20:46:31.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:31.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080590 -lt 777389080589 2026-02-22T20:46:31.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:46:31.306 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:46:31.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:46:31.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:46:31.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:46:31.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:46:31.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:46:31.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:46:31.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-22T20:46:31.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:46:31.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:46:31.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:46:31.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-22T20:46:31.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:31.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:46:31.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:31.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:46:31.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-22T20:46:28.066653+0000 2026-02-22T20:46:31.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-22T20:46:31.681 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-22T20:46:31.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-22T20:46:28.066653+0000 2026-02-22T20:46:31.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:46:31.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:46:28.066653+0000 2026-02-22T20:46:31.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:46:31.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:46:31.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:31.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:46:31.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:31.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:46:31.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:31.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:46:31.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:46:28.066653+0000 '>' 2026-02-22T20:46:28.066653+0000 2026-02-22T20:46:31.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:46:32.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:46:32.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:32.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:46:32.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:32.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:46:32.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:46:32.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:33.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:46:32.045091+0000 '>' 2026-02-22T20:46:28.066653+0000 2026-02-22T20:46:33.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:46:33.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6609: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:33.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6610: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6610: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6610: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:46:33.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6610: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:46:33.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6611: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:33.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6614: TEST_dual_store_replicated_cluster: pg_deep_scrub 3.0 2026-02-22T20:46:33.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-22T20:46:33.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-22T20:46:33.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:46:33.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:46:33.034 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:46:33.034 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:46:33.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:46:33.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:46:33.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:46:33.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:46:33.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:46:33.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:46:33.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:46:33.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:46:33.342 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:46:33.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:46:33.505 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:46:33.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:46:33.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:33.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:46:33.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407646 2026-02-22T20:46:33.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407646 2026-02-22T20:46:33.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407646' 2026-02-22T20:46:33.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:33.578 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:46:33.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080594 2026-02-22T20:46:33.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080594 2026-02-22T20:46:33.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407646 1-777389080594' 2026-02-22T20:46:33.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:33.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407646 2026-02-22T20:46:33.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:33.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:46:33.651 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407646 2026-02-22T20:46:33.651 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:33.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407646 2026-02-22T20:46:33.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407646' 2026-02-22T20:46:33.653 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 734439407646 2026-02-22T20:46:33.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:33.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407644 -lt 734439407646 2026-02-22T20:46:33.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:34.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:46:34.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:34.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407644 -lt 734439407646 2026-02-22T20:46:34.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:35.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:46:35.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:36.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407647 -lt 734439407646 2026-02-22T20:46:36.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:36.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080594 2026-02-22T20:46:36.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:36.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:46:36.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080594 2026-02-22T20:46:36.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:36.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080594 2026-02-22T20:46:36.140 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 777389080594 2026-02-22T20:46:36.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080594' 2026-02-22T20:46:36.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:36.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080595 -lt 777389080594 2026-02-22T20:46:36.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:46:36.306 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:46:36.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:46:36.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:46:36.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:46:36.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:46:36.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:46:36.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:46:36.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-22T20:46:36.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:46:36.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:46:36.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:46:36.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:46:36.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:36.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:46:36.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:36.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:46:36.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-22T20:44:27.072684+0000 2026-02-22T20:46:36.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-22T20:46:36.687 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-22T20:46:36.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-22T20:44:27.072684+0000 last_deep_scrub_stamp 2026-02-22T20:46:36.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:46:36.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:44:27.072684+0000 2026-02-22T20:46:36.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-22T20:46:36.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:46:36.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:36.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:46:36.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:36.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:46:36.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:36.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:46:36.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:44:27.072684+0000 '>' 2026-02-22T20:44:27.072684+0000 2026-02-22T20:46:36.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:46:37.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:46:37.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:37.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:46:37.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:37.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:46:37.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:46:37.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:38.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:44:27.072684+0000 '>' 2026-02-22T20:44:27.072684+0000 2026-02-22T20:46:38.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:46:39.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:46:39.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:39.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:46:39.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:39.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:46:39.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:39.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:46:39.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:46:37.025059+0000 '>' 2026-02-22T20:44:27.072684+0000 2026-02-22T20:46:39.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:46:39.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6615: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:39.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6616: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-22T20:46:39.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6616: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-22T20:46:39.176 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:46:39.176 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:46:39.176 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:46:39.176 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:46:39.176 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:46:39.177 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:46:39.177 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:46:39.177 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:46:39.177 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:46:39.177 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:46:39.177 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:46:39.177 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:46:39.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6616: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-22T20:46:39.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6616: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:46:39.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6617: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:39.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6620: TEST_dual_store_replicated_cluster: pg_scrub 3.0 2026-02-22T20:46:39.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-22T20:46:39.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-22T20:46:39.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:46:39.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:46:39.210 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:46:39.211 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:46:39.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:46:39.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:46:39.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:46:39.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:46:39.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:46:39.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:46:39.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:46:39.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:46:39.511 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:46:39.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:46:39.669 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:46:39.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:46:39.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:39.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:46:39.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407653 2026-02-22T20:46:39.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407653 2026-02-22T20:46:39.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407653' 2026-02-22T20:46:39.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:39.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:46:39.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080601 2026-02-22T20:46:39.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080601 2026-02-22T20:46:39.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407653 1-777389080601' 2026-02-22T20:46:39.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:39.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407653 2026-02-22T20:46:39.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:39.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:46:39.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407653 2026-02-22T20:46:39.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:39.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407653 2026-02-22T20:46:39.823 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 734439407653 2026-02-22T20:46:39.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407653' 2026-02-22T20:46:39.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:39.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407651 -lt 734439407653 2026-02-22T20:46:39.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:40.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:46:40.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:41.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407654 -lt 734439407653 2026-02-22T20:46:41.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:41.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080601 2026-02-22T20:46:41.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:41.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:46:41.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080601 2026-02-22T20:46:41.160 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:41.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080601 2026-02-22T20:46:41.162 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 777389080601 2026-02-22T20:46:41.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080601' 2026-02-22T20:46:41.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:41.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080602 -lt 777389080601 2026-02-22T20:46:41.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:46:41.328 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:46:41.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:46:41.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:46:41.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:46:41.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:46:41.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:46:41.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:46:41.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-22T20:46:41.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:46:41.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:46:41.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:46:41.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-22T20:46:41.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:41.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:46:41.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:41.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:46:41.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-22T20:46:37.025059+0000 2026-02-22T20:46:41.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-22T20:46:41.706 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-22T20:46:41.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-22T20:46:37.025059+0000 2026-02-22T20:46:41.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:46:41.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:46:37.025059+0000 2026-02-22T20:46:41.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:46:41.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:46:41.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:41.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:46:41.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:41.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:46:41.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:41.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:46:41.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:46:37.025059+0000 '>' 2026-02-22T20:46:37.025059+0000 2026-02-22T20:46:41.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:46:42.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:46:42.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:42.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:46:42.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:42.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:46:42.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:42.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:46:43.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:46:41.951836+0000 '>' 2026-02-22T20:46:37.025059+0000 2026-02-22T20:46:43.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:46:43.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6621: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:43.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6622: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-22T20:46:43.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6622: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6622: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:46:43.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6622: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:46:43.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6623: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:43.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6626: TEST_dual_store_replicated_cluster: diff -u td/osd-scrub-repair/dp_results.json td/osd-scrub-repair/sh2_results.json 2026-02-22T20:46:43.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6630: TEST_dual_store_replicated_cluster: inject_eio rep mdata csr_pool ROBJ13 td/osd-scrub-repair 1 2026-02-22T20:46:43.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2481: inject_eio: local pooltype=rep 2026-02-22T20:46:43.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2482: inject_eio: shift 2026-02-22T20:46:43.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2483: inject_eio: local which=mdata 2026-02-22T20:46:43.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2484: inject_eio: shift 2026-02-22T20:46:43.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2485: inject_eio: local poolname=csr_pool 2026-02-22T20:46:43.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2486: inject_eio: shift 2026-02-22T20:46:43.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2487: inject_eio: local objname=ROBJ13 2026-02-22T20:46:43.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2488: inject_eio: shift 2026-02-22T20:46:43.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2489: inject_eio: local dir=td/osd-scrub-repair 2026-02-22T20:46:43.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2490: inject_eio: shift 2026-02-22T20:46:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2491: inject_eio: local shard_id=1 2026-02-22T20:46:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2492: inject_eio: shift 2026-02-22T20:46:43.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: get_osds csr_pool ROBJ13 2026-02-22T20:46:43.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=csr_pool 2026-02-22T20:46:43.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=ROBJ13 2026-02-22T20:46:43.068 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map csr_pool ROBJ13 2026-02-22T20:46:43.069 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:46:43.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:46:43.227 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:46:43.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-22T20:46:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: initial_osds=('1' '0') 2026-02-22T20:46:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2494: inject_eio: local -a initial_osds 2026-02-22T20:46:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2495: inject_eio: local osd_id=0 2026-02-22T20:46:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2496: inject_eio: '[' rep '!=' ec ']' 2026-02-22T20:46:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2497: inject_eio: shard_id= 2026-02-22T20:46:43.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: cat td/osd-scrub-repair/0/type 2026-02-22T20:46:43.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2499: inject_eio: type=bluestore 2026-02-22T20:46:43.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2500: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2026-02-22T20:46:43.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2026-02-22T20:46:43.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=0 2026-02-22T20:46:43.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2026-02-22T20:46:43.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2026-02-22T20:46:43.229 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2026-02-22T20:46:43.230 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.0 2026-02-22T20:46:43.230 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:46:43.230 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:46:43.231 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:46:43.231 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:43.231 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:43.231 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:46:43.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.40073/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2026-02-22T20:46:43.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2026-02-22T20:46:43.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2501: inject_eio: local loop=0 2026-02-22T20:46:43.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2503: inject_eio: grep -q Invalid 2026-02-22T20:46:43.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: get_asok_path osd.0 2026-02-22T20:46:43.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T20:46:43.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T20:46:43.294 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:46:43.294 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:43.294 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:43.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-osd.0.asok 2026-02-22T20:46:43.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: CEPH_ARGS= 2026-02-22T20:46:43.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2502: inject_eio: ceph --admin-daemon /tmp/ceph-asok.40073/ceph-osd.0.asok injectmdataerr csr_pool ROBJ13 2026-02-22T20:46:43.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6632: TEST_dual_store_replicated_cluster: pg_deep_scrub 3.0 2026-02-22T20:46:43.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=3.0 2026-02-22T20:46:43.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 3.0 2026-02-22T20:46:43.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:46:43.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:46:43.356 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:46:43.356 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:46:43.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:46:43.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:46:43.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:46:43.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:46:43.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:46:43.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:46:43.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:46:43.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:46:43.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:46:43.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:46:43.823 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:46:43.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:46:43.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:43.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:46:43.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407658 2026-02-22T20:46:43.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407658 2026-02-22T20:46:43.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407658' 2026-02-22T20:46:43.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:43.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:46:43.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080606 2026-02-22T20:46:43.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080606 2026-02-22T20:46:43.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407658 1-777389080606' 2026-02-22T20:46:43.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:43.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407658 2026-02-22T20:46:43.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:43.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:46:43.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407658 2026-02-22T20:46:43.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:43.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407658 2026-02-22T20:46:43.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407658' 2026-02-22T20:46:43.973 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 734439407658 2026-02-22T20:46:43.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:44.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407656 -lt 734439407658 2026-02-22T20:46:44.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:45.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:46:45.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:45.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407659 -lt 734439407658 2026-02-22T20:46:45.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:45.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080606 2026-02-22T20:46:45.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:45.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:46:45.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080606 2026-02-22T20:46:45.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:45.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080606 2026-02-22T20:46:45.296 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 777389080606 2026-02-22T20:46:45.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080606' 2026-02-22T20:46:45.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:45.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080607 -lt 777389080606 2026-02-22T20:46:45.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:46:45.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:46:45.466 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:46:45.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:46:45.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:46:45.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:46:45.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:46:45.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:46:45.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-22T20:46:45.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:46:45.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:46:45.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:46:45.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:46:45.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:45.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:46:45.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:45.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:46:45.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-22T20:46:37.025059+0000 2026-02-22T20:46:45.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 3.0 2026-02-22T20:46:45.851 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0 on osd.1 to deep-scrub 2026-02-22T20:46:45.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 3.0 2026-02-22T20:46:37.025059+0000 last_deep_scrub_stamp 2026-02-22T20:46:45.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:46:45.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:46:37.025059+0000 2026-02-22T20:46:45.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-22T20:46:45.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:46:45.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:45.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:46:45.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:45.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:46:45.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:45.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:46:46.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:46:37.025059+0000 '>' 2026-02-22T20:46:37.025059+0000 2026-02-22T20:46:46.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:46:47.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:46:47.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:47.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_deep_scrub_stamp 2026-02-22T20:46:47.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:47.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T20:46:47.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:47.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_deep_scrub_stamp' 2026-02-22T20:46:47.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:46:46.082596+0000 '>' 2026-02-22T20:46:37.025059+0000 2026-02-22T20:46:47.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:46:47.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6634: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:47.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6635: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:47.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6637: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:47.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6641: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-22T20:46:47.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6641: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-22T20:46:47.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6641: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-22T20:46:47.180 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:46:47.180 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:46:47.180 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:46:47.180 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:46:47.181 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:46:47.181 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:46:47.181 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:46:47.181 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:46:47.181 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:46:47.181 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:46:47.181 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:46:47.181 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:46:47.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6642: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:46:47.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6644: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-22T20:46:47.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6644: TEST_dual_store_replicated_cluster: jq 'del(.inconsistents[] | select(.object.name == "ROBJ13"))' 2026-02-22T20:46:47.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6644: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-22T20:46:47.211 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:46:47.211 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:46:47.211 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:46:47.211 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:46:47.211 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:46:47.212 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:46:47.212 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:46:47.212 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:46:47.212 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:46:47.212 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:46:47.212 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:46:47.212 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:46:47.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6645: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-22T20:46:47.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6645: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:46:47.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6646: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:47.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6649: TEST_dual_store_replicated_cluster: pg_scrub 3.0 2026-02-22T20:46:47.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=3.0 2026-02-22T20:46:47.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 3.0 2026-02-22T20:46:47.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=3.0 2026-02-22T20:46:47.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:46:47.251 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:46:47.251 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:46:47.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:46:47.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:46:47.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:46:47.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:46:47.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:46:47.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:46:47.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:46:47.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:46:47.562 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:46:47.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:46:47.719 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:46:47.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:46:47.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:47.720 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:46:47.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=734439407663 2026-02-22T20:46:47.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 734439407663 2026-02-22T20:46:47.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407663' 2026-02-22T20:46:47.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:46:47.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:46:47.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=777389080611 2026-02-22T20:46:47.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 777389080611 2026-02-22T20:46:47.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-734439407663 1-777389080611' 2026-02-22T20:46:47.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:47.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-734439407663 2026-02-22T20:46:47.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:47.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:46:47.865 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-734439407663 2026-02-22T20:46:47.865 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:47.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=734439407663 2026-02-22T20:46:47.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 734439407663' 2026-02-22T20:46:47.867 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 734439407663 2026-02-22T20:46:47.867 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:48.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407661 -lt 734439407663 2026-02-22T20:46:48.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:46:49.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:46:49.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:46:49.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 734439407664 -lt 734439407663 2026-02-22T20:46:49.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:46:49.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-777389080611 2026-02-22T20:46:49.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:46:49.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:46:49.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-777389080611 2026-02-22T20:46:49.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:46:49.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=777389080611 2026-02-22T20:46:49.195 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 777389080611 2026-02-22T20:46:49.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 777389080611' 2026-02-22T20:46:49.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:46:49.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 777389080612 -lt 777389080611 2026-02-22T20:46:49.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:46:49.363 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 3.0 loop 0 2026-02-22T20:46:49.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 3.0 loop 0' 2026-02-22T20:46:49.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 3.0 2026-02-22T20:46:49.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=3.0 2026-02-22T20:46:49.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:46:49.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 3.0 query 2026-02-22T20:46:49.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:46:49.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+inconsistent 2026-02-22T20:46:49.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+inconsistent == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:46:49.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:46:49.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:46:49.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 3.0 2026-02-22T20:46:49.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:49.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:46:49.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:49.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:46:49.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-22T20:46:46.082596+0000 2026-02-22T20:46:49.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 3.0 2026-02-22T20:46:49.745 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 3.0 on osd.1 to scrub 2026-02-22T20:46:49.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 3.0 2026-02-22T20:46:46.082596+0000 2026-02-22T20:46:49.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=3.0 2026-02-22T20:46:49.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:46:46.082596+0000 2026-02-22T20:46:49.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:46:49.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:46:49.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:49.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:46:49.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:49.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:46:49.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:49.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:46:49.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:46:46.082596+0000 '>' 2026-02-22T20:46:46.082596+0000 2026-02-22T20:46:49.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:46:50.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:46:50.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:46:50.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 3.0 last_scrub_stamp 2026-02-22T20:46:50.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=3.0 2026-02-22T20:46:50.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:46:50.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="3.0") | .last_scrub_stamp' 2026-02-22T20:46:50.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:46:51.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:46:50.024922+0000 '>' 2026-02-22T20:46:46.082596+0000 2026-02-22T20:46:51.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:46:51.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6651: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:51.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6652: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:51.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6654: TEST_dual_store_replicated_cluster: (( extr_dbg >= 3 )) 2026-02-22T20:46:51.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6658: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-22T20:46:51.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6658: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-22T20:46:51.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6658: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-22T20:46:51.070 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:46:51.070 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:46:51.070 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:46:51.071 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:46:51.071 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:46:51.071 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:46:51.071 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:46:51.071 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:46:51.071 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:46:51.071 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:46:51.071 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:46:51.071 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:46:51.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6658: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:46:51.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6660: TEST_dual_store_replicated_cluster: rados list-inconsistent-obj 3.0 2026-02-22T20:46:51.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6660: TEST_dual_store_replicated_cluster: jq 'def walk(f): 2026-02-22T20:46:51.103 INFO:tasks.workunit.client.0.trial150.stderr: . as $in 2026-02-22T20:46:51.103 INFO:tasks.workunit.client.0.trial150.stderr: | if type == "object" then 2026-02-22T20:46:51.103 INFO:tasks.workunit.client.0.trial150.stderr: reduce keys[] as $key 2026-02-22T20:46:51.103 INFO:tasks.workunit.client.0.trial150.stderr: ( {}; . + { ($key): ($in[$key] | walk(f)) } ) | f 2026-02-22T20:46:51.103 INFO:tasks.workunit.client.0.trial150.stderr: elif type == "array" then map( walk(f) ) | f 2026-02-22T20:46:51.103 INFO:tasks.workunit.client.0.trial150.stderr: else f 2026-02-22T20:46:51.103 INFO:tasks.workunit.client.0.trial150.stderr: end; 2026-02-22T20:46:51.103 INFO:tasks.workunit.client.0.trial150.stderr:walk(if type == "object" then del(.mtime) else . end) 2026-02-22T20:46:51.103 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.local_mtime) else . end) 2026-02-22T20:46:51.103 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.last_reqid) else . end) 2026-02-22T20:46:51.103 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.version) else . end) 2026-02-22T20:46:51.103 INFO:tasks.workunit.client.0.trial150.stderr:| walk(if type == "object" then del(.prior_version) else . end)' 2026-02-22T20:46:51.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6660: TEST_dual_store_replicated_cluster: jq 'del(.inconsistents[] | select(.object.name == "ROBJ13"))' 2026-02-22T20:46:51.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6661: TEST_dual_store_replicated_cluster: jq .inconsistents 2026-02-22T20:46:51.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6661: TEST_dual_store_replicated_cluster: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print(json.dumps(ud, sort_keys=True, indent=2))' 2026-02-22T20:46:51.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6665: TEST_dual_store_replicated_cluster: diff -u td/osd-scrub-repair/dpPart2_w13_results.json td/osd-scrub-repair/sh2Part2_w13_results.json 2026-02-22T20:46:51.141 INFO:tasks.workunit.client.0.trial150.stdout:--- td/osd-scrub-repair/dpPart2_w13_results.json 2026-02-22 20:46:47.198660716 +0000 2026-02-22T20:46:51.141 INFO:tasks.workunit.client.0.trial150.stdout:+++ td/osd-scrub-repair/sh2Part2_w13_results.json 2026-02-22 20:46:51.090699825 +0000 2026-02-22T20:46:51.141 INFO:tasks.workunit.client.0.trial150.stdout:@@ -233,6 +233,38 @@ 2026-02-22T20:46:51.141 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:46:51.141 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head" 2026-02-22T20:46:51.141 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:46:51.141 INFO:tasks.workunit.client.0.trial150.stdout:+ "selected_object_info": { 2026-02-22T20:46:51.142 INFO:tasks.workunit.client.0.trial150.stdout:+ "alloc_hint_flags": 0, 2026-02-22T20:46:51.142 INFO:tasks.workunit.client.0.trial150.stdout:+ "data_digest": "0x2ddbf8f5", 2026-02-22T20:46:51.142 INFO:tasks.workunit.client.0.trial150.stdout:+ "expected_object_size": 0, 2026-02-22T20:46:51.142 INFO:tasks.workunit.client.0.trial150.stdout:+ "expected_write_size": 0, 2026-02-22T20:46:51.142 INFO:tasks.workunit.client.0.trial150.stdout:+ "flags": [ 2026-02-22T20:46:51.142 INFO:tasks.workunit.client.0.trial150.stdout:+ "dirty", 2026-02-22T20:46:51.142 INFO:tasks.workunit.client.0.trial150.stdout:+ "omap", 2026-02-22T20:46:51.142 INFO:tasks.workunit.client.0.trial150.stdout:+ "data_digest", 2026-02-22T20:46:51.142 INFO:tasks.workunit.client.0.trial150.stdout:+ "omap_digest" 2026-02-22T20:46:51.142 INFO:tasks.workunit.client.0.trial150.stdout:+ ], 2026-02-22T20:46:51.142 INFO:tasks.workunit.client.0.trial150.stdout:+ "lost": 0, 2026-02-22T20:46:51.143 INFO:tasks.workunit.client.0.trial150.stdout:+ "manifest": { 2026-02-22T20:46:51.143 INFO:tasks.workunit.client.0.trial150.stdout:+ "type": 0 2026-02-22T20:46:51.143 INFO:tasks.workunit.client.0.trial150.stdout:+ }, 2026-02-22T20:46:51.143 INFO:tasks.workunit.client.0.trial150.stdout:+ "oid": { 2026-02-22T20:46:51.143 INFO:tasks.workunit.client.0.trial150.stdout:+ "hash": 2682806379, 2026-02-22T20:46:51.143 INFO:tasks.workunit.client.0.trial150.stdout:+ "key": "", 2026-02-22T20:46:51.143 INFO:tasks.workunit.client.0.trial150.stdout:+ "max": 0, 2026-02-22T20:46:51.143 INFO:tasks.workunit.client.0.trial150.stdout:+ "namespace": "", 2026-02-22T20:46:51.143 INFO:tasks.workunit.client.0.trial150.stdout:+ "oid": "ROBJ13", 2026-02-22T20:46:51.143 INFO:tasks.workunit.client.0.trial150.stdout:+ "pool": 3, 2026-02-22T20:46:51.143 INFO:tasks.workunit.client.0.trial150.stdout:+ "snapid": -2 2026-02-22T20:46:51.143 INFO:tasks.workunit.client.0.trial150.stdout:+ }, 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout:+ "omap_digest": "0x6441854d", 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout:+ "shard_versions": [], 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout:+ "size": 7, 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout:+ "truncate_seq": 0, 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout:+ "truncate_size": 0, 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout:+ "user_version": 39, 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout:+ "watchers": {} 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout:+ }, 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout:@@ -242,17 +274,14 @@ 2026-02-22T20:46:51.144 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout:- "errors": [ 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout:- "read_error" 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout:- ], 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout:+ "errors": [], 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout: "size": 7 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [ 2026-02-22T20:46:51.145 INFO:tasks.workunit.client.0.trial150.stdout:- "stat_error", 2026-02-22T20:46:51.146 INFO:tasks.workunit.client.0.trial150.stdout:- "read_error" 2026-02-22T20:46:51.146 INFO:tasks.workunit.client.0.trial150.stdout:+ "stat_error" 2026-02-22T20:46:51.146 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:46:51.146 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:46:51.146 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:46:51.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6666: TEST_dual_store_replicated_cluster: diff -u td/osd-scrub-repair/dpPart2_wo13_results.json td/osd-scrub-repair/shPart2_wo13_results.json 2026-02-22T20:46:51.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6668: TEST_dual_store_replicated_cluster: ceph osd pool rm csr_pool csr_pool --yes-i-really-really-mean-it 2026-02-22T20:46:51.351 INFO:tasks.workunit.client.0.trial150.stderr:pool 'csr_pool' removed 2026-02-22T20:46:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6669: TEST_dual_store_replicated_cluster: return 0 2026-02-22T20:46:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:46:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:46:51.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:46:51.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:46:51.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:46:51.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:46:51.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:46:51.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:46:51.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:46:51.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:46:51.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:46:51.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:46:51.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:46:51.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:46:51.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:46:51.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:46:51.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:46:51.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:46:51.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:46:51.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:46:51.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:46:51.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:46:51.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:46:51.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:46:51.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:51.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:51.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:46:51.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:46:51.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:46:51.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:46:51.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:46:51.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:46:51.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:46:51.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:46:51.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:46:51.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:46:51.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:46:51.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:46:51.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:46:51.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:46:51.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:46:51.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:46:51.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:46:51.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:46:51.518 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:46:51.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:46:51.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:46:51.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:46:51.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:46:51.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:46:51.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:46:51.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:46:51.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:46:51.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:46:51.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:46:51.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:46:51.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:51.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:51.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:46:51.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:46:51.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:46:51.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:46:51.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:46:51.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:51.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:51.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:46:51.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:46:51.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:46:51.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:46:51.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:46:51.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_appends td/osd-scrub-repair 2026-02-22T20:46:51.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1037: TEST_list_missing_erasure_coded_appends: list_missing_erasure_coded td/osd-scrub-repair false 2026-02-22T20:46:51.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:975: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2026-02-22T20:46:51.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:976: list_missing_erasure_coded: local allow_overwrites=false 2026-02-22T20:46:51.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:977: list_missing_erasure_coded: local poolname=ecpool 2026-02-22T20:46:51.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:979: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-22T20:46:51.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:46:51.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:46:51.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:46:51.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:46:51.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:46:51.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:46:51.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:46:51.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:46:51.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:46:51.555 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:46:51.555 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:51.555 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:51.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:51.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:46:51.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:46:51.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:46:51.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:46:51.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:46:51.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:46:51.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:46:51.587 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:46:51.587 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:46:51.587 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:46:51.587 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:46:51.587 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:51.587 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:51.588 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:46:51.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:46:51.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:46:51.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:46:51.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:46:51.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:46:51.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:46:51.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:46:51.646 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:46:51.646 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:46:51.646 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:46:51.646 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:46:51.646 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:51.646 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:51.646 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:46:51.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:46:51.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:46:51.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:980: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-22T20:46:51.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:46:51.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:46:51.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:46:51.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:46:51.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:46:51.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:46:51.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:46:51.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:46:51.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:46:51.801 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:46:51.801 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:51.801 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:51.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:51.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:46:51.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:46:51.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: seq 0 2 2026-02-22T20:46:51.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:46:51.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2026-02-22T20:46:51.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:46:51.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:46:51.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:46:51.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:46:51.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:46:51.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:46:51.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:46:51.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:46:51.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:46:51.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:46:51.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:46:51.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:46:51.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:46:51.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:46:51.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:46:51.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:46:51.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:46:51.824 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:46:51.824 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:51.825 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:51.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:51.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:51.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:46:51.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:46:51.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:46:51.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:46:51.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:46:51.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:46:51.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:46:51.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:46:51.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:46:51.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:46:51.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:46:51.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:46:51.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:46:51.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=aafd0b83-387f-4b7c-8d03-8bcd65e494e1 2026-02-22T20:46:51.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 aafd0b83-387f-4b7c-8d03-8bcd65e494e1' 2026-02-22T20:46:51.829 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 aafd0b83-387f-4b7c-8d03-8bcd65e494e1 2026-02-22T20:46:51.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:46:51.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA7a5tpW+hcMRAA6yoeL2Ze6dVlVSOfIhBl9A== 2026-02-22T20:46:51.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA7a5tpW+hcMRAA6yoeL2Ze6dVlVSOfIhBl9A=="}' 2026-02-22T20:46:51.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new aafd0b83-387f-4b7c-8d03-8bcd65e494e1 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:46:51.927 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:46:51.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:46:51.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA7a5tpW+hcMRAA6yoeL2Ze6dVlVSOfIhBl9A== --osd-uuid aafd0b83-387f-4b7c-8d03-8bcd65e494e1 2026-02-22T20:46:51.952 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:51.942+0000 7f32540cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:51.954 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:51.942+0000 7f32540cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:51.955 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:51.942+0000 7f32540cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:51.955 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:51.942+0000 7f32540cba40 -1 bdev(0x561fbad10c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:46:51.955 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:51.942+0000 7f32540cba40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:46:52.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:46:52.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:46:52.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:46:52.744 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:46:52.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:46:52.946 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:46:52.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:46:52.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:46:52.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:46:52.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:46:52.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:46:52.962 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:52.950+0000 7f198058ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:52.964 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:52.954+0000 7f198058ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:52.965 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:52.954+0000 7f198058ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:53.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:46:53.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:46:53.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:46:53.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:46:53.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:46:53.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:53.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:46:53.109 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:46:53.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:53.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:46:53.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:46:53.353 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:53.342+0000 7f198058ea40 -1 Falling back to public interface 2026-02-22T20:46:53.911 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:53.898+0000 7f198058ea40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:46:54.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:46:54.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:54.270 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:46:54.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:46:54.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:54.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:46:54.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:46:55.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:46:55.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:55.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:46:55.436 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:46:55.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:55.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:46:55.597 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2101768079,v1:127.0.0.1:6803/2101768079] [v2:127.0.0.1:6804/2101768079,v1:127.0.0.1:6805/2101768079] exists,up aafd0b83-387f-4b7c-8d03-8bcd65e494e1 2026-02-22T20:46:55.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:46:55.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:46:55.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:46:55.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:46:55.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2026-02-22T20:46:55.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:46:55.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:46:55.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:46:55.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:46:55.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:46:55.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:46:55.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:46:55.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:46:55.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:46:55.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:46:55.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:46:55.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:46:55.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:46:55.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:46:55.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:46:55.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:46:55.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:46:55.603 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:46:55.603 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:55.603 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:55.603 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:55.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:55.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:46:55.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:46:55.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:46:55.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:46:55.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:46:55.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:46:55.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:46:55.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:46:55.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:46:55.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:46:55.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:46:55.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:46:55.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:46:55.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0bb364f6-452b-4708-943c-1fe733410b5a 2026-02-22T20:46:55.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 0bb364f6-452b-4708-943c-1fe733410b5a' 2026-02-22T20:46:55.605 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 0bb364f6-452b-4708-943c-1fe733410b5a 2026-02-22T20:46:55.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:46:55.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA/a5tpbewaJBAAPYayk4LRD0xBd6t5ZH6buQ== 2026-02-22T20:46:55.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA/a5tpbewaJBAAPYayk4LRD0xBd6t5ZH6buQ=="}' 2026-02-22T20:46:55.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0bb364f6-452b-4708-943c-1fe733410b5a -i td/osd-scrub-repair/1/new.json 2026-02-22T20:46:55.771 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:46:55.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:46:55.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/a5tpbewaJBAAPYayk4LRD0xBd6t5ZH6buQ== --osd-uuid 0bb364f6-452b-4708-943c-1fe733410b5a 2026-02-22T20:46:55.798 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:55.786+0000 7f7c50698a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:55.800 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:55.790+0000 7f7c50698a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:55.801 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:55.790+0000 7f7c50698a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:55.801 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:55.790+0000 7f7c50698a40 -1 bdev(0x5589596f4c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:46:55.802 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:55.790+0000 7f7c50698a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:46:56.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:46:56.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:46:56.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:46:56.440 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:46:56.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:46:56.645 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:46:56.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:46:56.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:46:56.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:46:56.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:46:56.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:46:56.670 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:56.658+0000 7f0abaae6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:56.672 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:56.662+0000 7f0abaae6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:56.673 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:56.662+0000 7f0abaae6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:56.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:46:56.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:46:56.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:46:56.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:46:56.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:46:56.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:56.807 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:46:56.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:46:56.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:56.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:46:56.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:46:57.033 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:57.022+0000 7f0abaae6a40 -1 Falling back to public interface 2026-02-22T20:46:57.622 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:57.610+0000 7f0abaae6a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:46:57.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:46:57.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:57.971 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:46:57.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:46:57.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:57.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:46:58.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:46:59.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:46:59.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:46:59.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:46:59.134 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:46:59.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:46:59.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:46:59.295 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1159018592,v1:127.0.0.1:6811/1159018592] [v2:127.0.0.1:6812/1159018592,v1:127.0.0.1:6813/1159018592] exists,up 0bb364f6-452b-4708-943c-1fe733410b5a 2026-02-22T20:46:59.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:46:59.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:46:59.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:46:59.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:46:59.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2026-02-22T20:46:59.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:46:59.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:46:59.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:46:59.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:46:59.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:46:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:46:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:46:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:46:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:46:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:46:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:46:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:46:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:46:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:46:59.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:46:59.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:46:59.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:46:59.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:46:59.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:46:59.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:46:59.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:46:59.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:46:59.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:46:59.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4e4ce862-bfa0-486b-b1dc-5d8e14101d91 2026-02-22T20:46:59.303 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 4e4ce862-bfa0-486b-b1dc-5d8e14101d91 2026-02-22T20:46:59.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 4e4ce862-bfa0-486b-b1dc-5d8e14101d91' 2026-02-22T20:46:59.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:46:59.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBDa5tphkAfEhAA6bcOBxZhl9J44/bw77zb/A== 2026-02-22T20:46:59.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBDa5tphkAfEhAA6bcOBxZhl9J44/bw77zb/A=="}' 2026-02-22T20:46:59.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4e4ce862-bfa0-486b-b1dc-5d8e14101d91 -i td/osd-scrub-repair/2/new.json 2026-02-22T20:46:59.476 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:46:59.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:46:59.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDa5tphkAfEhAA6bcOBxZhl9J44/bw77zb/A== --osd-uuid 4e4ce862-bfa0-486b-b1dc-5d8e14101d91 2026-02-22T20:46:59.504 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:59.494+0000 7f3560955a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:59.506 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:59.494+0000 7f3560955a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:59.507 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:59.494+0000 7f3560955a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:46:59.507 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:59.494+0000 7f3560955a40 -1 bdev(0x56130c72cc00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:46:59.507 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:46:59.494+0000 7f3560955a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:47:00.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:47:00.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:47:00.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:47:00.236 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:47:00.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:47:00.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:47:00.442 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:47:00.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:47:00.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:47:00.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:47:00.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:47:00.466 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:00.454+0000 7f18e0ca8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:00.468 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:00.458+0000 7f18e0ca8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:00.469 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:00.458+0000 7f18e0ca8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:00.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:47:00.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:47:00.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:47:00.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:47:00.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:47:00.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:00.608 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:47:00.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:47:00.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:00.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:47:00.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:00.849 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:00.838+0000 7f18e0ca8a40 -1 Falling back to public interface 2026-02-22T20:47:01.417 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:01.406+0000 7f18e0ca8a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:47:01.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:01.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:01.767 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:47:01.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:47:01.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:01.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:47:01.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:02.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:02.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:02.929 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:47:02.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:47:02.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:02.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:47:03.090 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/987859491,v1:127.0.0.1:6819/987859491] [v2:127.0.0.1:6820/987859491,v1:127.0.0.1:6821/987859491] exists,up 4e4ce862-bfa0-486b-b1dc-5d8e14101d91 2026-02-22T20:47:03.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:47:03.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:47:03.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:47:03.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:984: list_missing_erasure_coded: create_rbd_pool 2026-02-22T20:47:03.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:47:03.238 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:47:03.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:47:03.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:47:03.467 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:47:03.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:47:04.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:47:04.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: wait_for_clean 2026-02-22T20:47:04.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:47:04.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:47:04.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:47:04.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:47:04.756 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:47:04.756 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:47:04.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:47:04.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:47:04.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:47:04.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:47:04.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:47:04.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:47:04.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:47:04.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:47:04.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:47:05.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:47:05.047 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:05.047 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:47:05.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:47:05.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:05.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:47:05.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-22T20:47:05.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-22T20:47:05.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T20:47:05.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:05.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:47:05.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-22T20:47:05.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-22T20:47:05.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2026-02-22T20:47:05.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:05.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:47:05.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-22T20:47:05.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-22T20:47:05.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-60129542146' 2026-02-22T20:47:05.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:05.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-22T20:47:05.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:05.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:47:05.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-22T20:47:05.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:05.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-22T20:47:05.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T20:47:05.274 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836484 2026-02-22T20:47:05.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:47:05.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-22T20:47:05.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:47:06.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:47:06.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:47:06.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-22T20:47:06.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:06.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-22T20:47:06.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:06.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:47:06.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-22T20:47:06.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:06.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-22T20:47:06.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-22T20:47:06.610 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672963 2026-02-22T20:47:06.610 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:47:06.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-22T20:47:06.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:06.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-22T20:47:06.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:06.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:47:06.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-22T20:47:06.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:06.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-22T20:47:06.781 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542146 2026-02-22T20:47:06.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-22T20:47:06.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:47:06.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-22T20:47:06.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:47:06.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:47:06.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:47:07.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:47:07.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:47:07.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:47:07.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:47:07.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:47:07.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:47:07.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:47:07.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:47:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:47:07.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:47:07.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:47:07.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:47:07.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:47:07.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:47:07.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:47:07.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:987: list_missing_erasure_coded: create_ec_pool ecpool false k=2 m=1 2026-02-22T20:47:07.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-22T20:47:07.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:47:07.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-22T20:47:07.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:47:07.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-22T20:47:07.697 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:47:07.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-22T20:47:07.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-22T20:47:08.001 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' created 2026-02-22T20:47:08.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:47:09.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-22T20:47:09.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:47:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:47:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:47:09.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:47:09.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:47:09.017 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:47:09.017 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:47:09.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:47:09.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:47:09.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:47:09.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:47:09.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:47:09.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:47:09.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:47:09.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:47:09.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:47:09.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:47:09.294 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:09.294 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:47:09.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:47:09.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:09.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:47:09.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-22T20:47:09.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-22T20:47:09.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-22T20:47:09.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:09.368 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:47:09.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672965 2026-02-22T20:47:09.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672965 2026-02-22T20:47:09.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965' 2026-02-22T20:47:09.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:09.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:47:09.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-22T20:47:09.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-22T20:47:09.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-42949672965 2-60129542148' 2026-02-22T20:47:09.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:09.511 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-22T20:47:09.511 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:09.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:47:09.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-22T20:47:09.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:09.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-22T20:47:09.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-22T20:47:09.515 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836485 2026-02-22T20:47:09.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:47:09.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-22T20:47:09.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:47:10.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:47:10.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:47:10.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-22T20:47:10.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:10.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672965 2026-02-22T20:47:10.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:10.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:47:10.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672965 2026-02-22T20:47:10.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:10.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672965 2026-02-22T20:47:10.853 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672965 2026-02-22T20:47:10.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2026-02-22T20:47:10.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:47:11.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672965 2026-02-22T20:47:11.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:11.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-22T20:47:11.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:11.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:47:11.024 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-22T20:47:11.024 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:11.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-22T20:47:11.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-22T20:47:11.026 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542148 2026-02-22T20:47:11.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:47:11.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-22T20:47:11.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:47:11.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:47:11.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:47:11.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:47:11.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:47:11.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:47:11.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:47:11.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:47:11.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:47:11.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:47:11.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:47:11.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:47:11.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:47:11.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:47:11.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:47:11.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:47:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:47:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:47:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:47:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:990: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2026-02-22T20:47:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:47:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:47:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2026-02-22T20:47:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:47:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:47:11.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:47:11.974 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:47:11.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:47:12.175 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:47:12.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:47:12.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:47:12.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:47:12.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: get_osds ecpool MOBJ0 2026-02-22T20:47:12.213 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-22T20:47:12.213 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ0 2026-02-22T20:47:12.213 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ0 2026-02-22T20:47:12.213 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:47:12.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-22T20:47:12.381 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:12.381 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:47:12.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2026-02-22T20:47:12.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: osds0=('2' '1' '0') 2026-02-22T20:47:12.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: local -a osds0 2026-02-22T20:47:12.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:994: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2026-02-22T20:47:12.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:47:12.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:47:12.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2026-02-22T20:47:12.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:47:12.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:47:12.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:47:12.582 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:47:12.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:47:12.789 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:47:12.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:47:12.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:47:12.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:47:12.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: get_osds ecpool MOBJ1 2026-02-22T20:47:12.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-22T20:47:12.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ1 2026-02-22T20:47:12.820 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ1 2026-02-22T20:47:12.821 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:47:12.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-22T20:47:12.981 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:12.981 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:47:12.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2026-02-22T20:47:12.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: osds1=('2' '1' '0') 2026-02-22T20:47:12.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: local -a osds1 2026-02-22T20:47:12.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: seq 0 2 2026-02-22T20:47:12.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:12.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:47:12.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:47:12.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:47:12.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:47:12.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:47:12.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:47:13.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:47:13.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:13.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:47:13.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:47:13.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:47:13.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:47:13.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:47:13.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:47:13.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:47:13.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:13.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-22T20:47:13.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:47:13.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:47:13.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:47:13.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:47:13.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:47:13.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:47:13.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: list_missing_erasure_coded: id=2 2026-02-22T20:47:13.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2026-02-22T20:47:14.113 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#2:a9030505:::MOBJ0:head# 2026-02-22T20:47:14.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1005: list_missing_erasure_coded: id=1 2026-02-22T20:47:14.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2026-02-22T20:47:14.416 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#2:a9030505:::MOBJ0:head# 2026-02-22T20:47:14.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1009: list_missing_erasure_coded: id=1 2026-02-22T20:47:14.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2026-02-22T20:47:14.706 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2026-02-22T20:47:14.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1012: list_missing_erasure_coded: id=0 2026-02-22T20:47:14.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1013: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2026-02-22T20:47:15.016 INFO:tasks.workunit.client.0.trial150.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2026-02-22T20:47:15.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: seq 0 2 2026-02-22T20:47:15.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:15.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2026-02-22T20:47:15.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:47:15.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:47:15.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:47:15.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:47:15.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:47:15.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:47:15.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:47:15.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:47:15.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:47:15.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:47:15.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:47:15.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:47:15.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:47:15.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:47:15.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:47:15.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:47:15.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:47:15.133 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:47:15.133 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:15.133 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:15.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:15.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:15.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:47:15.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:47:15.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:47:15.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:47:15.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:47:15.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:47:15.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:47:15.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:47:15.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:47:15.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:47:15.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:47:15.136 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:47:15.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:47:15.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:47:15.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:47:15.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:47:15.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:47:15.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:47:15.158 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:15.146+0000 7f559b2fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:15.159 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:15.150+0000 7f559b2fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:15.160 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:15.150+0000 7f559b2fea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:15.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:47:15.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:47:15.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:47:15.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:47:15.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:47:15.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:15.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:47:15.296 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:47:15.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:15.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:47:15.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:15.569 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:15.558+0000 7f559b2fea40 -1 Falling back to public interface 2026-02-22T20:47:16.150 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:16.138+0000 7f559b2fea40 -1 osd.0 31 log_to_monitors true 2026-02-22T20:47:16.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:16.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:16.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:47:16.457 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:16.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:16.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:47:16.618 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 32 last_clean_interval [5,31) [v2:127.0.0.1:6802/1524931799,v1:127.0.0.1:6803/1524931799] [v2:127.0.0.1:6804/1524931799,v1:127.0.0.1:6805/1524931799] exists,up aafd0b83-387f-4b7c-8d03-8bcd65e494e1 2026-02-22T20:47:16.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:47:16.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:47:16.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:47:16.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:16.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2026-02-22T20:47:16.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:47:16.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:47:16.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:47:16.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:47:16.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:47:16.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:47:16.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:47:16.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:47:16.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:47:16.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:47:16.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:47:16.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:47:16.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:47:16.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:47:16.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:47:16.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:47:16.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:47:16.620 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:47:16.620 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:16.620 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:16.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:16.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:16.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:47:16.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:47:16.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:47:16.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:47:16.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:47:16.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:47:16.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:47:16.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:47:16.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:47:16.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:47:16.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:47:16.623 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:47:16.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:47:16.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:47:16.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:47:16.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:47:16.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:47:16.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:47:16.636 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:16.626+0000 7f4055b42a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:16.638 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:16.626+0000 7f4055b42a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:16.639 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:16.626+0000 7f4055b42a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:16.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:47:16.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:47:16.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:47:16.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:47:16.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:47:16.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:16.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:47:16.785 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:47:16.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:16.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:47:16.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:16.985 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:16.974+0000 7f4055b42a40 -1 Falling back to public interface 2026-02-22T20:47:17.530 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:17.518+0000 7f4055b42a40 -1 osd.1 33 log_to_monitors true 2026-02-22T20:47:17.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:17.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:17.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:47:17.949 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:17.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:17.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:47:18.109 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 34 last_clean_interval [10,33) [v2:127.0.0.1:6810/73714431,v1:127.0.0.1:6811/73714431] [v2:127.0.0.1:6812/73714431,v1:127.0.0.1:6813/73714431] exists,up 0bb364f6-452b-4708-943c-1fe733410b5a 2026-02-22T20:47:18.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:47:18.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:47:18.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:47:18.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:18.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2026-02-22T20:47:18.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:47:18.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:47:18.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:47:18.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:47:18.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:47:18.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:47:18.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:47:18.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:47:18.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:47:18.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:47:18.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:47:18.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:47:18.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:47:18.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:47:18.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:47:18.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:47:18.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:47:18.111 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:47:18.111 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:18.111 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:18.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:18.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:18.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:47:18.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:47:18.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:47:18.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:47:18.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:47:18.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:47:18.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:47:18.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:47:18.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:47:18.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:47:18.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:47:18.114 INFO:tasks.workunit.client.0.trial150.stderr:start osd.2 2026-02-22T20:47:18.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:47:18.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-22T20:47:18.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:47:18.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:47:18.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:47:18.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:47:18.128 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:18.118+0000 7fa215a0fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:18.129 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:18.118+0000 7fa215a0fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:18.130 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:18.118+0000 7fa215a0fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:18.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:47:18.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:47:18.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:47:18.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:47:18.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:47:18.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:18.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:47:18.274 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:47:18.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:18.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:47:18.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:18.516 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:18.506+0000 7fa215a0fa40 -1 Falling back to public interface 2026-02-22T20:47:19.141 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:19.130+0000 7fa215a0fa40 -1 osd.2 35 log_to_monitors true 2026-02-22T20:47:19.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:19.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:19.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:47:19.434 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:19.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:19.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:47:19.591 INFO:tasks.workunit.client.0.trial150.stderr:osd.2 up in weight 1 up_from 46 up_thru 46 down_at 36 last_clean_interval [14,35) [v2:127.0.0.1:6818/777599466,v1:127.0.0.1:6819/777599466] [v2:127.0.0.1:6820/777599466,v1:127.0.0.1:6821/777599466] exists,up 4e4ce862-bfa0-486b-b1dc-5d8e14101d91 2026-02-22T20:47:19.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:47:19.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:47:19.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:47:19.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: list_missing_erasure_coded: create_rbd_pool 2026-02-22T20:47:19.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:47:19.791 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' removed 2026-02-22T20:47:19.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:47:19.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:47:19.998 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:47:20.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:47:21.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:47:21.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: list_missing_erasure_coded: wait_for_clean 2026-02-22T20:47:21.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:47:21.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:47:21.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:47:21.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:47:21.294 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:47:21.294 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:47:21.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:47:21.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:47:21.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:47:21.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:47:21.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:47:21.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:47:21.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:47:21.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:47:21.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:47:21.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:47:21.579 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:21.579 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:47:21.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:47:21.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:21.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:47:21.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=163208757251 2026-02-22T20:47:21.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 163208757251 2026-02-22T20:47:21.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757251' 2026-02-22T20:47:21.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:21.652 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:47:21.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=176093659138 2026-02-22T20:47:21.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 176093659138 2026-02-22T20:47:21.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757251 1-176093659138' 2026-02-22T20:47:21.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:21.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:47:21.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=197568495618 2026-02-22T20:47:21.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 197568495618 2026-02-22T20:47:21.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-163208757251 1-176093659138 2-197568495618' 2026-02-22T20:47:21.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:21.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-163208757251 2026-02-22T20:47:21.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:21.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:47:21.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-163208757251 2026-02-22T20:47:21.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:21.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=163208757251 2026-02-22T20:47:21.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 163208757251' 2026-02-22T20:47:21.803 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 163208757251 2026-02-22T20:47:21.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:47:21.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 163208757251 2026-02-22T20:47:21.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:47:22.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:47:22.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:47:23.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 163208757251 -lt 163208757251 2026-02-22T20:47:23.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:23.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-176093659138 2026-02-22T20:47:23.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:23.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:47:23.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-176093659138 2026-02-22T20:47:23.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:23.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=176093659138 2026-02-22T20:47:23.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 176093659138' 2026-02-22T20:47:23.142 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 176093659138 2026-02-22T20:47:23.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:47:23.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 176093659138 -lt 176093659138 2026-02-22T20:47:23.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:23.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-197568495618 2026-02-22T20:47:23.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:23.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:47:23.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-197568495618 2026-02-22T20:47:23.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:23.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=197568495618 2026-02-22T20:47:23.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 197568495618' 2026-02-22T20:47:23.319 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 197568495618 2026-02-22T20:47:23.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:47:23.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 197568495618 -lt 197568495618 2026-02-22T20:47:23.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:47:23.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:47:23.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:47:23.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:47:23.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:47:23.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:47:23.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:47:23.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:47:23.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:47:23.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:47:23.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:47:23.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:47:23.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:47:23.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:47:23.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:47:24.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:47:24.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:47:24.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:47:24.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: list_missing_erasure_coded: get_pg ecpool MOBJ0 2026-02-22T20:47:24.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:47:24.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=MOBJ0 2026-02-22T20:47:24.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool MOBJ0 2026-02-22T20:47:24.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:47:24.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: list_missing_erasure_coded: local pg=2.0 2026-02-22T20:47:24.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: list_missing_erasure_coded: repair 2.0 2026-02-22T20:47:24.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:47:24.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:47:24.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:47:24.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:47:24.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:47:24.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:47:24.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:47:07.992981+0000 2026-02-22T20:47:24.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:47:24.508 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.2 to repair 2026-02-22T20:47:24.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:47:07.992981+0000 2026-02-22T20:47:24.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:47:24.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:47:07.992981+0000 2026-02-22T20:47:24.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:47:24.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:47:24.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:47:24.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:47:24.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:47:24.520 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:47:24.520 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:47:24.520 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:47:24.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:47:07.992981+0000 '>' 2026-02-22T20:47:07.992981+0000 2026-02-22T20:47:24.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:47:25.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:47:25.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:47:25.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:47:25.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:47:25.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:47:25.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:47:25.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:47:25.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:47:07.992981+0000 '>' 2026-02-22T20:47:07.992981+0000 2026-02-22T20:47:25.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:47:26.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:47:26.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:47:26.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:47:26.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:47:26.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:47:26.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:47:26.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:47:26.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:47:07.992981+0000 '>' 2026-02-22T20:47:07.992981+0000 2026-02-22T20:47:26.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:47:27.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:47:27.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:47:27.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:47:27.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:47:27.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:47:27.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:47:27.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:47:28.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:47:07.992981+0000 '>' 2026-02-22T20:47:07.992981+0000 2026-02-22T20:47:28.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:47:29.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:47:29.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:47:29.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:47:29.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:47:29.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:47:29.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:47:29.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:47:29.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:47:07.992981+0000 '>' 2026-02-22T20:47:07.992981+0000 2026-02-22T20:47:29.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:47:30.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:47:30.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:47:30.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:47:30.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:47:30.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:47:30.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:47:30.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:47:30.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:47:25.112094+0000 '>' 2026-02-22T20:47:07.992981+0000 2026-02-22T20:47:30.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:47:30.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: list_missing_erasure_coded: seq 0 120 2026-02-22T20:47:30.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: list_missing_erasure_coded: for i in $(seq 0 120) 2026-02-22T20:47:30.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2026-02-22T20:47:30.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2026-02-22T20:47:30.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2026-02-22T20:47:30.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: wc -l 2026-02-22T20:47:30.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: matches=2 2026-02-22T20:47:30.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2026-02-22T20:47:30.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: list_missing_erasure_coded: break 2026-02-22T20:47:30.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:47:30.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:47:30.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:47:30.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:47:30.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:47:30.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:47:30.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:47:30.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:47:30.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:47:30.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:47:30.666 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:47:30.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:47:30.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:47:30.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:47:30.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:47:30.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:47:30.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:47:30.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:47:30.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:47:30.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:47:30.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:47:30.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:47:30.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:47:30.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:47:30.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:30.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:30.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:47:30.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:47:30.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:47:30.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:47:30.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:47:30.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:47:30.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:47:30.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:47:30.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:47:30.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:47:30.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:47:30.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:47:30.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:47:30.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:47:30.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:47:30.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:47:30.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:47:30.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:47:30.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:47:30.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:47:30.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:47:30.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:47:30.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:47:30.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:47:30.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:47:30.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:47:30.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:47:30.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:47:30.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:47:30.704 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:47:30.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:30.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:30.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:47:30.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:47:30.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:47:30.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:47:30.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:47:30.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:30.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:30.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:47:30.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:47:30.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:47:30.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:47:30.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:47:30.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_list_missing_erasure_coded_overwrites td/osd-scrub-repair 2026-02-22T20:47:30.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1041: TEST_list_missing_erasure_coded_overwrites: '[' true = true ']' 2026-02-22T20:47:30.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1042: TEST_list_missing_erasure_coded_overwrites: list_missing_erasure_coded td/osd-scrub-repair true 2026-02-22T20:47:30.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:975: list_missing_erasure_coded: local dir=td/osd-scrub-repair 2026-02-22T20:47:30.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:976: list_missing_erasure_coded: local allow_overwrites=true 2026-02-22T20:47:30.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:977: list_missing_erasure_coded: local poolname=ecpool 2026-02-22T20:47:30.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:979: list_missing_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-22T20:47:30.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:47:30.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:47:30.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:47:30.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:47:30.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:47:30.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:47:30.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:47:30.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:47:30.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:47:30.733 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:47:30.734 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:30.734 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:30.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:30.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:47:30.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:47:30.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:47:30.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:47:30.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:47:30.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:47:30.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:47:30.764 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:47:30.765 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:47:30.765 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:47:30.765 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:47:30.765 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:30.765 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:30.765 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:47:30.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:47:30.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:47:30.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:47:30.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:47:30.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:47:30.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:47:30.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:47:30.821 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:47:30.821 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:47:30.821 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:47:30.821 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:47:30.821 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:30.821 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:30.822 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:47:30.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:47:30.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:47:30.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:980: list_missing_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-22T20:47:30.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:47:30.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:47:30.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:47:30.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:47:30.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:47:30.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:47:30.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:47:30.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:47:30.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:47:30.974 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:47:30.975 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:30.975 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:30.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:30.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:47:30.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:47:30.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: seq 0 2 2026-02-22T20:47:30.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:30.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 0 2026-02-22T20:47:30.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:47:30.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:47:30.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:47:30.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:47:30.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:47:30.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:47:30.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:47:30.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:47:31.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:47:31.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:47:31.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:47:31.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:47:31.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:47:31.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:47:31.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:47:31.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:47:31.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:47:31.000 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:47:31.000 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:31.000 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:31.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:31.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:31.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:47:31.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:47:31.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:47:31.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:47:31.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:47:31.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:47:31.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:47:31.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:47:31.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:47:31.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:47:31.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:47:31.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:47:31.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:47:31.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=be9371da-97d1-4fa4-83ba-5dae06aa6063 2026-02-22T20:47:31.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 be9371da-97d1-4fa4-83ba-5dae06aa6063' 2026-02-22T20:47:31.003 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 be9371da-97d1-4fa4-83ba-5dae06aa6063 2026-02-22T20:47:31.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:47:31.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBja5tpnHNCABAAKdY9VPr9UGxuQ6pwmZ7vqg== 2026-02-22T20:47:31.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBja5tpnHNCABAAKdY9VPr9UGxuQ6pwmZ7vqg=="}' 2026-02-22T20:47:31.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new be9371da-97d1-4fa4-83ba-5dae06aa6063 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:47:31.101 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:47:31.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:47:31.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBja5tpnHNCABAAKdY9VPr9UGxuQ6pwmZ7vqg== --osd-uuid be9371da-97d1-4fa4-83ba-5dae06aa6063 2026-02-22T20:47:31.127 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:31.115+0000 7feaad9d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:31.128 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:31.119+0000 7feaad9d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:31.129 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:31.119+0000 7feaad9d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:31.129 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:31.119+0000 7feaad9d8a40 -1 bdev(0x562e2fc7ac00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:47:31.129 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:31.119+0000 7feaad9d8a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:47:31.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:47:31.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:47:31.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:47:31.916 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:47:31.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:47:32.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:47:32.118 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:47:32.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:47:32.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:47:32.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:47:32.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:47:32.142 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:32.131+0000 7f85d9f3da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:32.144 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:32.135+0000 7f85d9f3da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:32.145 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:32.135+0000 7f85d9f3da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:32.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:47:32.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:47:32.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:47:32.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:47:32.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:47:32.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:32.282 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:47:32.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:47:32.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:32.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:47:32.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:32.573 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:32.563+0000 7f85d9f3da40 -1 Falling back to public interface 2026-02-22T20:47:33.143 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:33.131+0000 7f85d9f3da40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:47:33.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:33.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:33.444 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:47:33.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:47:33.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:33.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:47:33.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:34.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:34.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:34.602 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:47:34.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:47:34.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:34.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:47:34.759 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/12365358,v1:127.0.0.1:6803/12365358] [v2:127.0.0.1:6804/12365358,v1:127.0.0.1:6805/12365358] exists,up be9371da-97d1-4fa4-83ba-5dae06aa6063 2026-02-22T20:47:34.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:47:34.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:47:34.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:47:34.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:34.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 1 2026-02-22T20:47:34.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:47:34.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:47:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:47:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:47:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:47:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:47:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:47:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:47:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:47:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:47:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:47:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:47:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:47:34.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:47:34.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:47:34.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:47:34.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:47:34.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:47:34.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:47:34.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:47:34.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:47:34.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:47:34.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:47:34.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d8f7df6f-3e7c-403b-98b6-f3fdfd37e50c 2026-02-22T20:47:34.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d8f7df6f-3e7c-403b-98b6-f3fdfd37e50c' 2026-02-22T20:47:34.766 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 d8f7df6f-3e7c-403b-98b6-f3fdfd37e50c 2026-02-22T20:47:34.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:47:34.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBma5tpbRO+LRAAD0nvpUGebgP8++2IaFFuOQ== 2026-02-22T20:47:34.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBma5tpbRO+LRAAD0nvpUGebgP8++2IaFFuOQ=="}' 2026-02-22T20:47:34.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d8f7df6f-3e7c-403b-98b6-f3fdfd37e50c -i td/osd-scrub-repair/1/new.json 2026-02-22T20:47:34.927 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:47:34.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:47:34.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBma5tpbRO+LRAAD0nvpUGebgP8++2IaFFuOQ== --osd-uuid d8f7df6f-3e7c-403b-98b6-f3fdfd37e50c 2026-02-22T20:47:34.954 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:34.943+0000 7f2453d0fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:34.955 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:34.947+0000 7f2453d0fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:34.956 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:34.947+0000 7f2453d0fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:34.957 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:34.947+0000 7f2453d0fa40 -1 bdev(0x5651374d2c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:47:34.957 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:34.947+0000 7f2453d0fa40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:47:35.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:47:35.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:47:35.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:47:35.595 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:47:35.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:47:35.791 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:47:35.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:47:35.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:47:35.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:47:35.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:47:35.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:47:35.815 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:35.807+0000 7f362c643a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:35.817 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:35.807+0000 7f362c643a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:35.818 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:35.807+0000 7f362c643a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:35.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:47:35.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:47:35.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:47:35.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:47:35.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:47:35.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:35.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:47:35.954 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:47:35.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:35.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:47:36.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:36.237 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:36.227+0000 7f362c643a40 -1 Falling back to public interface 2026-02-22T20:47:36.802 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:36.791+0000 7f362c643a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:47:37.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:37.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:37.113 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:47:37.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:47:37.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:37.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:47:37.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:38.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:38.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:38.277 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:47:38.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:47:38.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:38.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:47:38.435 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3645355794,v1:127.0.0.1:6811/3645355794] [v2:127.0.0.1:6812/3645355794,v1:127.0.0.1:6813/3645355794] exists,up d8f7df6f-3e7c-403b-98b6-f3fdfd37e50c 2026-02-22T20:47:38.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:47:38.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:47:38.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:47:38.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:981: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:38.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:982: list_missing_erasure_coded: run_osd td/osd-scrub-repair 2 2026-02-22T20:47:38.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:47:38.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:47:38.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:47:38.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:47:38.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:47:38.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:47:38.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:47:38.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:47:38.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:47:38.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:47:38.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:47:38.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:47:38.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:47:38.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:47:38.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:47:38.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:47:38.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:47:38.441 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:47:38.441 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:38.441 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:38.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:38.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:38.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:47:38.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:47:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:47:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:47:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:47:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:47:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:47:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:47:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:47:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:47:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:47:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:47:38.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:47:38.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e272ee2a-1e72-4ede-91a4-ac67d732df26 2026-02-22T20:47:38.443 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 e272ee2a-1e72-4ede-91a4-ac67d732df26 2026-02-22T20:47:38.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 e272ee2a-1e72-4ede-91a4-ac67d732df26' 2026-02-22T20:47:38.444 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:47:38.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBqa5tpS/+SGhAA0xjE6hTq4klyBTcjPlWAbQ== 2026-02-22T20:47:38.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBqa5tpS/+SGhAA0xjE6hTq4klyBTcjPlWAbQ=="}' 2026-02-22T20:47:38.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e272ee2a-1e72-4ede-91a4-ac67d732df26 -i td/osd-scrub-repair/2/new.json 2026-02-22T20:47:38.615 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:47:38.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:47:38.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqa5tpS/+SGhAA0xjE6hTq4klyBTcjPlWAbQ== --osd-uuid e272ee2a-1e72-4ede-91a4-ac67d732df26 2026-02-22T20:47:38.641 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:38.631+0000 7fe6b171ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:38.642 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:38.631+0000 7fe6b171ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:38.644 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:38.635+0000 7fe6b171ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:38.644 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:38.635+0000 7fe6b171ea40 -1 bdev(0x563c19bc4c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:47:38.644 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:38.635+0000 7fe6b171ea40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:47:39.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:47:39.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:47:39.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:47:39.259 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:47:39.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:47:39.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:47:39.464 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:47:39.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:47:39.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:47:39.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:47:39.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:47:39.487 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:39.475+0000 7f67ba8c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:39.488 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:39.479+0000 7f67ba8c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:39.490 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:39.479+0000 7f67ba8c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:39.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:47:39.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:47:39.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:47:39.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:47:39.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:47:39.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:39.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:47:39.626 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:47:39.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:39.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:47:39.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:39.853 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:39.843+0000 7f67ba8c4a40 -1 Falling back to public interface 2026-02-22T20:47:40.546 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:40.535+0000 7f67ba8c4a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:47:40.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:40.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:40.787 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:47:40.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:47:40.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:40.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:47:40.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:41.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:41.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:41.948 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:47:41.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:47:41.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:41.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:47:42.113 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3090948501,v1:127.0.0.1:6819/3090948501] [v2:127.0.0.1:6820/3090948501,v1:127.0.0.1:6821/3090948501] exists,up e272ee2a-1e72-4ede-91a4-ac67d732df26 2026-02-22T20:47:42.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:47:42.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:47:42.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:47:42.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:984: list_missing_erasure_coded: create_rbd_pool 2026-02-22T20:47:42.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:47:42.261 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:47:42.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:47:42.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:47:42.475 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:47:42.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:47:43.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:47:43.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:985: list_missing_erasure_coded: wait_for_clean 2026-02-22T20:47:43.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:47:43.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:47:43.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:47:43.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:47:43.770 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:47:43.770 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:47:43.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:47:43.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:47:43.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:47:43.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:47:43.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:47:43.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:47:43.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:47:43.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:47:43.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:47:44.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:47:44.053 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:44.053 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:47:44.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:47:44.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:44.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:47:44.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:47:44.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:47:44.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:47:44.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:44.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:47:44.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-22T20:47:44.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-22T20:47:44.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-22T20:47:44.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:44.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:47:44.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-22T20:47:44.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-22T20:47:44.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705667 2-60129542146' 2026-02-22T20:47:44.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:44.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:47:44.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:44.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:47:44.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:47:44.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:44.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:47:44.276 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:47:44.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:47:44.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:47:44.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:47:44.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:47:45.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:47:45.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:47:45.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-22T20:47:45.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:45.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-22T20:47:45.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:45.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:47:45.597 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705667 2026-02-22T20:47:45.597 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:45.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705667 2026-02-22T20:47:45.599 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705667 2026-02-22T20:47:45.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-22T20:47:45.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:47:45.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-22T20:47:45.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:45.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-22T20:47:45.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:45.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:47:45.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-22T20:47:45.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:45.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-22T20:47:45.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-22T20:47:45.773 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542146 2026-02-22T20:47:45.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:47:45.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-22T20:47:45.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:47:45.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:47:45.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:47:46.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:47:46.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:47:46.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:47:46.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:47:46.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:47:46.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:47:46.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:47:46.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:47:46.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:47:46.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:47:46.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:47:46.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:47:46.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:47:46.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:47:46.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:47:46.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:987: list_missing_erasure_coded: create_ec_pool ecpool true k=2 m=1 2026-02-22T20:47:46.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-22T20:47:46.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:47:46.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-22T20:47:46.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:47:46.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-22T20:47:46.684 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:47:46.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-22T20:47:46.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-22T20:47:46.989 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' created 2026-02-22T20:47:47.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:47:48.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-22T20:47:48.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-22T20:47:48.205 INFO:tasks.workunit.client.0.trial150.stderr:set pool 2 allow_ec_overwrites to true 2026-02-22T20:47:48.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:47:48.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:47:48.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:47:48.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:47:48.218 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:47:48.219 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:47:48.219 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:47:48.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:47:48.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:47:48.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:47:48.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:47:48.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:47:48.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:47:48.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:47:48.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:47:48.348 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:47:48.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:47:48.512 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:48.512 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:47:48.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:47:48.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:48.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:47:48.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-22T20:47:48.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-22T20:47:48.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485' 2026-02-22T20:47:48.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:48.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:47:48.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-22T20:47:48.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-22T20:47:48.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669' 2026-02-22T20:47:48.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:47:48.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:47:48.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-22T20:47:48.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-22T20:47:48.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836485 1-38654705669 2-60129542148' 2026-02-22T20:47:48.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:48.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-22T20:47:48.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:48.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:47:48.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836485 2026-02-22T20:47:48.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:48.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836485 2026-02-22T20:47:48.735 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836485 2026-02-22T20:47:48.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-22T20:47:48.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:47:48.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836485 2026-02-22T20:47:48.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:47:49.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:47:49.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:47:50.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836485 2026-02-22T20:47:50.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:50.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-22T20:47:50.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:50.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:47:50.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-22T20:47:50.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:50.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-22T20:47:50.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-22T20:47:50.065 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705669 2026-02-22T20:47:50.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:47:50.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705669 -lt 38654705669 2026-02-22T20:47:50.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:47:50.223 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-22T20:47:50.224 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:47:50.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:47:50.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-22T20:47:50.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:47:50.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-22T20:47:50.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-22T20:47:50.228 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542148 2026-02-22T20:47:50.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:47:50.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542148 2026-02-22T20:47:50.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:47:50.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:47:50.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:47:50.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:47:50.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:47:50.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:47:50.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:47:50.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:47:50.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:47:50.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:47:50.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:47:50.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:47:50.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:47:50.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:47:50.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:47:50.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:47:50.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:47:50.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:47:50.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:47:50.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:990: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ0 2026-02-22T20:47:50.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:47:50.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:47:50.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ0 2026-02-22T20:47:50.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:47:50.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:47:50.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:47:51.181 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:47:51.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:47:51.382 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:47:51.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:47:51.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:47:51.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ0 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:47:51.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: get_osds ecpool MOBJ0 2026-02-22T20:47:51.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-22T20:47:51.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ0 2026-02-22T20:47:51.414 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ0 2026-02-22T20:47:51.414 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:47:51.574 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-22T20:47:51.574 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:51.574 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:47:51.574 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2026-02-22T20:47:51.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: osds0=('2' '1' '0') 2026-02-22T20:47:51.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:991: list_missing_erasure_coded: local -a osds0 2026-02-22T20:47:51.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:994: list_missing_erasure_coded: add_something td/osd-scrub-repair ecpool MOBJ1 2026-02-22T20:47:51.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:47:51.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:47:51.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=MOBJ1 2026-02-22T20:47:51.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:47:51.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:47:51.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:47:51.780 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:47:51.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:47:51.985 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:47:51.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:47:51.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:47:51.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put MOBJ1 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:47:52.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: get_osds ecpool MOBJ1 2026-02-22T20:47:52.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-22T20:47:52.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=MOBJ1 2026-02-22T20:47:52.019 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool MOBJ1 2026-02-22T20:47:52.019 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:47:52.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=2 2026-02-22T20:47:52.177 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:52.177 INFO:tasks.workunit.client.0.trial150.stderr:0' 2026-02-22T20:47:52.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 2 1 0 2026-02-22T20:47:52.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: osds1=('2' '1' '0') 2026-02-22T20:47:52.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:995: list_missing_erasure_coded: local -a osds1 2026-02-22T20:47:52.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: seq 0 2 2026-02-22T20:47:52.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:52.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:47:52.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:47:52.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:47:52.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:47:52.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:47:52.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:47:52.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:47:52.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:52.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:47:52.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:47:52.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:47:52.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:47:52.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:47:52.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:47:52.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:47:52.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:998: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:52.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:999: list_missing_erasure_coded: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-22T20:47:52.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:47:52.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:47:52.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:47:52.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:47:52.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:47:53.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:47:53.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1002: list_missing_erasure_coded: id=2 2026-02-22T20:47:53.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1003: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 MOBJ0 remove 2026-02-22T20:47:53.283 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#2:a9030505:::MOBJ0:head# 2026-02-22T20:47:53.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1005: list_missing_erasure_coded: id=1 2026-02-22T20:47:53.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1006: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ0 remove 2026-02-22T20:47:53.588 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#2:a9030505:::MOBJ0:head# 2026-02-22T20:47:53.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1009: list_missing_erasure_coded: id=1 2026-02-22T20:47:53.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1010: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 MOBJ1 remove 2026-02-22T20:47:53.874 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#2:8dd16d55:::MOBJ1:head# 2026-02-22T20:47:53.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1012: list_missing_erasure_coded: id=0 2026-02-22T20:47:53.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1013: list_missing_erasure_coded: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 MOBJ1 remove 2026-02-22T20:47:54.200 INFO:tasks.workunit.client.0.trial150.stdout:remove 2#2:8dd16d55:::MOBJ1:head# 2026-02-22T20:47:54.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: seq 0 2 2026-02-22T20:47:54.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:54.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 0 2026-02-22T20:47:54.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:47:54.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:47:54.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:47:54.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:47:54.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:47:54.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:47:54.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:47:54.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:47:54.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:47:54.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:47:54.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:47:54.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:47:54.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:47:54.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:47:54.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:47:54.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:47:54.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:47:54.304 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:47:54.304 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:54.304 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:54.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:54.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:54.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:47:54.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:47:54.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:47:54.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:47:54.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:47:54.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:47:54.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:47:54.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:47:54.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:47:54.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:47:54.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:47:54.307 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:47:54.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:47:54.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:47:54.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:47:54.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:47:54.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:47:54.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:47:54.329 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:54.319+0000 7f1cc39d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:54.330 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:54.319+0000 7f1cc39d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:54.332 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:54.323+0000 7f1cc39d2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:54.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:47:54.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:47:54.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:47:54.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:47:54.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:47:54.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:54.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:47:54.468 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:47:54.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:54.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:47:54.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:54.736 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:54.727+0000 7f1cc39d2a40 -1 Falling back to public interface 2026-02-22T20:47:55.312 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:55.303+0000 7f1cc39d2a40 -1 osd.0 32 log_to_monitors true 2026-02-22T20:47:55.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:55.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:55.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:47:55.625 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:55.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:55.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:47:55.782 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 39 up_thru 39 down_at 33 last_clean_interval [5,32) [v2:127.0.0.1:6802/2824710080,v1:127.0.0.1:6803/2824710080] [v2:127.0.0.1:6804/2824710080,v1:127.0.0.1:6805/2824710080] exists,up be9371da-97d1-4fa4-83ba-5dae06aa6063 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 1 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:47:55.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:47:55.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:47:55.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:47:55.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:47:55.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:47:55.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:47:55.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:47:55.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:47:55.784 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:47:55.784 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:55.784 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:55.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:55.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:55.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:47:55.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:47:55.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:47:55.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:47:55.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:47:55.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:47:55.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:47:55.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:47:55.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:47:55.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:47:55.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:47:55.786 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T20:47:55.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:47:55.787 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:47:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:47:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:47:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:47:55.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:47:55.804 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:55.795+0000 7f6f4b39ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:55.806 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:55.795+0000 7f6f4b39ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:55.807 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:55.799+0000 7f6f4b39ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:55.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:47:55.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:47:55.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:47:55.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:47:55.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:47:55.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:55.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:47:55.953 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:47:55.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:55.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:47:56.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:56.220 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:56.211+0000 7f6f4b39ea40 -1 Falling back to public interface 2026-02-22T20:47:56.827 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:56.815+0000 7f6f4b39ea40 -1 osd.1 34 log_to_monitors true 2026-02-22T20:47:57.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:57.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:57.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:47:57.111 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:57.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:57.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:47:57.267 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 35 last_clean_interval [9,34) [v2:127.0.0.1:6810/3989182679,v1:127.0.0.1:6811/3989182679] [v2:127.0.0.1:6812/3989182679,v1:127.0.0.1:6813/3989182679] exists,up d8f7df6f-3e7c-403b-98b6-f3fdfd37e50c 2026-02-22T20:47:57.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:47:57.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:47:57.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:47:57.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1016: list_missing_erasure_coded: for id in $(seq 0 2) 2026-02-22T20:47:57.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1017: list_missing_erasure_coded: activate_osd td/osd-scrub-repair 2 2026-02-22T20:47:57.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:47:57.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:47:57.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:47:57.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:47:57.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:47:57.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:47:57.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:47:57.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:47:57.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:47:57.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:47:57.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:47:57.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:47:57.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:47:57.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:47:57.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:47:57.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:47:57.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:47:57.269 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:47:57.269 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:47:57.270 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:47:57.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:57.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:47:57.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:47:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:47:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:47:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:47:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:47:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:47:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:47:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:47:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:47:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:47:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:47:57.272 INFO:tasks.workunit.client.0.trial150.stderr:start osd.2 2026-02-22T20:47:57.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:47:57.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-22T20:47:57.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:47:57.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:47:57.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:47:57.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:47:57.294 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:57.283+0000 7f035126ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:57.296 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:57.287+0000 7f035126ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:57.296 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:57.287+0000 7f035126ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:47:57.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:47:57.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:47:57.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:47:57.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:47:57.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:47:57.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:57.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:47:57.432 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:47:57.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:57.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:47:57.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:47:57.645 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:57.635+0000 7f035126ea40 -1 Falling back to public interface 2026-02-22T20:47:58.226 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:47:58.215+0000 7f035126ea40 -1 osd.2 36 log_to_monitors true 2026-02-22T20:47:58.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:47:58.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:47:58.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:47:58.591 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:47:58.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:47:58.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:47:58.749 INFO:tasks.workunit.client.0.trial150.stderr:osd.2 up in weight 1 up_from 47 up_thru 47 down_at 37 last_clean_interval [14,36) [v2:127.0.0.1:6818/791676327,v1:127.0.0.1:6819/791676327] [v2:127.0.0.1:6820/791676327,v1:127.0.0.1:6821/791676327] exists,up e272ee2a-1e72-4ede-91a4-ac67d732df26 2026-02-22T20:47:58.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:47:58.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:47:58.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:47:58.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1019: list_missing_erasure_coded: create_rbd_pool 2026-02-22T20:47:58.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:47:58.949 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' removed 2026-02-22T20:47:58.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:47:58.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:47:59.155 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:47:59.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:48:00.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:48:00.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1020: list_missing_erasure_coded: wait_for_clean 2026-02-22T20:48:00.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:48:00.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:48:00.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:48:00.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:48:00.447 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:48:00.447 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:48:00.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:48:00.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:48:00.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:48:00.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:48:00.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:48:00.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:48:00.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:48:00.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:48:00.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:48:00.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:48:00.736 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:48:00.736 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:48:00.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:48:00.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:00.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:48:00.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724547 2026-02-22T20:48:00.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724547 2026-02-22T20:48:00.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724547' 2026-02-22T20:48:00.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:00.810 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:48:00.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=184683593730 2026-02-22T20:48:00.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 184683593730 2026-02-22T20:48:00.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724547 1-184683593730' 2026-02-22T20:48:00.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:00.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:48:00.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=201863462914 2026-02-22T20:48:00.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 201863462914 2026-02-22T20:48:00.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-167503724547 1-184683593730 2-201863462914' 2026-02-22T20:48:00.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:00.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-167503724547 2026-02-22T20:48:00.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:00.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:48:00.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-167503724547 2026-02-22T20:48:00.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:00.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724547 2026-02-22T20:48:00.961 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 167503724547 2026-02-22T20:48:00.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 167503724547' 2026-02-22T20:48:00.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:01.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 167503724547 2026-02-22T20:48:01.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:48:02.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:48:02.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:02.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724547 -lt 167503724547 2026-02-22T20:48:02.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:02.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-184683593730 2026-02-22T20:48:02.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:02.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:48:02.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-184683593730 2026-02-22T20:48:02.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:02.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=184683593730 2026-02-22T20:48:02.292 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 184683593730 2026-02-22T20:48:02.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 184683593730' 2026-02-22T20:48:02.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:48:02.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 184683593730 -lt 184683593730 2026-02-22T20:48:02.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:02.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-201863462914 2026-02-22T20:48:02.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:02.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:48:02.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-201863462914 2026-02-22T20:48:02.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:02.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=201863462914 2026-02-22T20:48:02.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 201863462914' 2026-02-22T20:48:02.466 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 201863462914 2026-02-22T20:48:02.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:48:02.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 201863462914 -lt 201863462914 2026-02-22T20:48:02.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:48:02.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:48:02.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:48:02.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:48:02.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:48:02.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:48:02.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:48:02.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:48:02.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:48:02.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:48:02.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:48:02.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:48:02.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:48:02.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:48:02.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:48:03.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:48:03.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:48:03.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:48:03.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: list_missing_erasure_coded: get_pg ecpool MOBJ0 2026-02-22T20:48:03.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:48:03.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=MOBJ0 2026-02-22T20:48:03.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool MOBJ0 2026-02-22T20:48:03.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:48:03.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1023: list_missing_erasure_coded: local pg=2.0 2026-02-22T20:48:03.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1027: list_missing_erasure_coded: repair 2.0 2026-02-22T20:48:03.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=2.0 2026-02-22T20:48:03.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 2.0 2026-02-22T20:48:03.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:48:03.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:48:03.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:48:03.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:48:03.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:47:46.981149+0000 2026-02-22T20:48:03.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 2.0 2026-02-22T20:48:03.643 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 2.0s0 on osd.2 to repair 2026-02-22T20:48:03.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 2.0 2026-02-22T20:47:46.981149+0000 2026-02-22T20:48:03.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=2.0 2026-02-22T20:48:03.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:47:46.981149+0000 2026-02-22T20:48:03.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:48:03.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:48:03.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:48:03.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:48:03.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:48:03.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:48:03.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:48:03.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:48:03.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:47:46.981149+0000 '>' 2026-02-22T20:47:46.981149+0000 2026-02-22T20:48:03.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:48:04.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:48:04.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:48:04.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:48:04.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:48:04.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:48:04.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:48:04.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:48:04.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:47:46.981149+0000 '>' 2026-02-22T20:47:46.981149+0000 2026-02-22T20:48:04.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:48:05.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:48:05.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:48:05.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:48:05.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:48:05.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:48:05.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:48:05.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:48:06.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:47:46.981149+0000 '>' 2026-02-22T20:47:46.981149+0000 2026-02-22T20:48:06.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:48:07.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:48:07.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:48:07.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:48:07.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:48:07.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:48:07.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:48:07.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:48:07.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:47:46.981149+0000 '>' 2026-02-22T20:47:46.981149+0000 2026-02-22T20:48:07.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:48:08.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:48:08.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:48:08.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:48:08.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:48:08.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:48:08.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:48:08.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:48:08.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:47:46.981149+0000 '>' 2026-02-22T20:47:46.981149+0000 2026-02-22T20:48:08.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:48:09.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:48:09.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:48:09.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 2.0 last_scrub_stamp 2026-02-22T20:48:09.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=2.0 2026-02-22T20:48:09.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:48:09.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .last_scrub_stamp' 2026-02-22T20:48:09.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:48:09.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:48:04.180956+0000 '>' 2026-02-22T20:47:46.981149+0000 2026-02-22T20:48:09.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:48:09.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: list_missing_erasure_coded: seq 0 120 2026-02-22T20:48:09.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1029: list_missing_erasure_coded: for i in $(seq 0 120) 2026-02-22T20:48:09.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1030: list_missing_erasure_coded: '[' 0 -lt 60 ']' 2026-02-22T20:48:09.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: ceph pg 2.0 list_unfound 2026-02-22T20:48:09.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: egrep 'MOBJ0|MOBJ1' 2026-02-22T20:48:09.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: wc -l 2026-02-22T20:48:09.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1031: list_missing_erasure_coded: matches=2 2026-02-22T20:48:09.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: list_missing_erasure_coded: '[' 2 -eq 2 ']' 2026-02-22T20:48:09.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:1032: list_missing_erasure_coded: break 2026-02-22T20:48:09.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:48:09.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:48:09.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:48:09.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:48:09.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:48:09.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:48:09.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:48:09.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:48:09.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:48:09.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:48:09.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:48:09.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:48:09.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:48:09.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:48:09.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:48:09.808 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:48:09.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:48:09.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:48:09.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:48:09.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:48:09.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:48:09.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:48:09.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:48:09.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:48:09.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:09.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:09.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:48:09.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:48:09.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:48:09.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:48:09.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:48:09.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:48:09.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:48:09.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:48:09.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:48:09.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:48:09.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:48:09.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:48:09.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:48:09.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:48:09.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:48:09.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:48:09.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:48:09.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:48:09.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:48:09.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:48:09.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:48:09.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:48:09.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:48:09.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:48:09.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:48:09.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:48:09.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:48:09.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:48:09.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:48:09.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:48:09.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:09.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:09.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:48:09.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:48:09.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:48:09.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:48:09.845 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:48:09.845 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:09.845 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:09.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:48:09.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:48:09.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:48:09.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:48:09.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:48:09.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_periodic_scrub_replicated td/osd-scrub-repair 2026-02-22T20:48:09.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5926: TEST_periodic_scrub_replicated: local dir=td/osd-scrub-repair 2026-02-22T20:48:09.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5927: TEST_periodic_scrub_replicated: local poolname=psr_pool 2026-02-22T20:48:09.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5928: TEST_periodic_scrub_replicated: local objname=POBJ 2026-02-22T20:48:09.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:5929: TEST_periodic_scrub_replicated: return 0 2026-02-22T20:48:09.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:48:09.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:48:09.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:48:09.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:48:09.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:48:09.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:48:09.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:48:09.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:48:09.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:48:09.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:48:09.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:48:09.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:48:09.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:48:09.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:48:09.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:48:09.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:48:09.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:48:09.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:48:09.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:48:09.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:48:09.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:48:09.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:48:09.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:48:09.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:48:09.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:09.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:09.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:48:09.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:48:09.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:48:09.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:48:09.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:48:09.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:48:09.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:48:09.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:48:09.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:48:09.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:48:09.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:48:09.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:48:09.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:48:09.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:48:09.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:48:09.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:48:09.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:48:09.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:48:09.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:48:09.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:48:09.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:48:09.871 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:48:09.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:48:09.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:48:09.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:48:09.873 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:48:09.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:48:09.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:48:09.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:48:09.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:48:09.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:09.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:09.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:48:09.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:48:09.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:48:09.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:48:09.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:48:09.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:09.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:09.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:48:09.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:48:09.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:48:09.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:48:09.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:48:09.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats td/osd-scrub-repair 2026-02-22T20:48:09.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:713: TEST_repair_stats: local dir=td/osd-scrub-repair 2026-02-22T20:48:09.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:714: TEST_repair_stats: local poolname=testpool 2026-02-22T20:48:09.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:715: TEST_repair_stats: local OSDS=2 2026-02-22T20:48:09.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:716: TEST_repair_stats: local OBJS=30 2026-02-22T20:48:09.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:718: TEST_repair_stats: local REPAIRS=20 2026-02-22T20:48:09.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:721: TEST_repair_stats: run_mon td/osd-scrub-repair a 2026-02-22T20:48:09.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:48:09.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:48:09.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:48:09.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:48:09.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:48:09.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:48:09.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:48:09.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:48:09.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:48:09.907 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:48:09.907 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:09.907 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:09.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:09.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:48:09.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:48:09.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:48:09.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:48:09.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:48:09.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:48:09.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:48:09.938 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:48:09.938 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:48:09.938 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:48:09.939 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:48:09.939 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:09.939 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:09.939 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:48:09.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:48:09.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:48:09.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:48:09.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:48:09.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:48:09.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:48:09.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:48:09.998 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:48:09.998 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:48:09.998 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:48:09.998 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:48:09.998 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:09.998 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:09.998 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:48:09.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:48:09.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:48:10.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:722: TEST_repair_stats: run_mgr td/osd-scrub-repair x 2026-02-22T20:48:10.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:48:10.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:48:10.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:48:10.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:48:10.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:48:10.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:48:10.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:48:10.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:48:10.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:48:10.149 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:48:10.149 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:10.149 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:10.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:10.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:48:10.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:48:10.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:724: TEST_repair_stats: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:48:10.172 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats: expr 2 - 1 2026-02-22T20:48:10.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats: seq 0 1 2026-02-22T20:48:10.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:48:10.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:10.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:48:10.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:48:10.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:48:10.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:48:10.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:10.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:48:10.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:48:10.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:48:10.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:48:10.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:48:10.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:48:10.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:48:10.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:48:10.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:48:10.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:48:10.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:48:10.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:48:10.183 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:48:10.183 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:10.184 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:10.184 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:10.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:10.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:48:10.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:48:10.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:48:10.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:48:10.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:48:10.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:48:10.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:48:10.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:48:10.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:48:10.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:48:10.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:48:10.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:48:10.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:48:10.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=55b3afd0-21a2-4bc8-a7e8-e9f19f0e44d1 2026-02-22T20:48:10.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 55b3afd0-21a2-4bc8-a7e8-e9f19f0e44d1' 2026-02-22T20:48:10.187 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 55b3afd0-21a2-4bc8-a7e8-e9f19f0e44d1 2026-02-22T20:48:10.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:48:10.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCKa5tpfan4ChAASgL+d1jEzZWcjKpe8GxKYg== 2026-02-22T20:48:10.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCKa5tpfan4ChAASgL+d1jEzZWcjKpe8GxKYg=="}' 2026-02-22T20:48:10.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 55b3afd0-21a2-4bc8-a7e8-e9f19f0e44d1 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:48:10.281 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:48:10.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:48:10.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCKa5tpfan4ChAASgL+d1jEzZWcjKpe8GxKYg== --osd-uuid 55b3afd0-21a2-4bc8-a7e8-e9f19f0e44d1 2026-02-22T20:48:10.306 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:10.295+0000 7f9078ba6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:10.308 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:10.299+0000 7f9078ba6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:10.308 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:10.299+0000 7f9078ba6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:10.309 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:10.299+0000 7f9078ba6a40 -1 bdev(0x564e3a5e8c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:48:10.309 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:10.299+0000 7f9078ba6a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:48:11.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:48:11.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:48:11.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:48:11.040 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:48:11.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:48:11.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:48:11.247 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:48:11.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:11.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:48:11.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:48:11.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:48:11.273 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:11.263+0000 7ff93e1c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:11.275 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:11.267+0000 7ff93e1c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:11.276 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:11.267+0000 7ff93e1c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:11.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:48:11.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:48:11.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:48:11.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:48:11.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:48:11.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:11.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:48:11.406 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:48:11.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:11.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:48:11.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:48:11.664 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:11.655+0000 7ff93e1c5a40 -1 Falling back to public interface 2026-02-22T20:48:12.200 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:12.191+0000 7ff93e1c5a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:48:12.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:48:12.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:12.565 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:48:12.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:48:12.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:12.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:48:12.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:48:13.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:48:13.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:13.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:48:13.739 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:48:13.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:13.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:48:13.896 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1749845588,v1:127.0.0.1:6803/1749845588] [v2:127.0.0.1:6804/1749845588,v1:127.0.0.1:6805/1749845588] exists,up 55b3afd0-21a2-4bc8-a7e8-e9f19f0e44d1 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:725: TEST_repair_stats: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:726: TEST_repair_stats: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:48:13.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:48:13.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:48:13.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:48:13.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:48:13.898 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:48:13.898 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:13.898 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:13.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:48:13.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:48:13.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:48:13.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=45c02d4d-58fb-43c7-821c-b2d7e1a47d10 2026-02-22T20:48:13.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 45c02d4d-58fb-43c7-821c-b2d7e1a47d10' 2026-02-22T20:48:13.902 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 45c02d4d-58fb-43c7-821c-b2d7e1a47d10 2026-02-22T20:48:13.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:48:13.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCNa5tpk1X/NRAASIs14d07KPgyf3DpulqvwQ== 2026-02-22T20:48:13.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCNa5tpk1X/NRAASIs14d07KPgyf3DpulqvwQ=="}' 2026-02-22T20:48:13.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 45c02d4d-58fb-43c7-821c-b2d7e1a47d10 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:48:14.082 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:48:14.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:48:14.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCNa5tpk1X/NRAASIs14d07KPgyf3DpulqvwQ== --osd-uuid 45c02d4d-58fb-43c7-821c-b2d7e1a47d10 2026-02-22T20:48:14.110 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:14.099+0000 7efed89a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:14.112 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:14.103+0000 7efed89a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:14.113 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:14.103+0000 7efed89a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:14.113 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:14.103+0000 7efed89a8a40 -1 bdev(0x560eda4f4c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:48:14.113 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:14.103+0000 7efed89a8a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:48:14.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:48:14.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:48:14.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:48:14.796 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:48:14.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:48:14.996 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:48:14.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:48:14.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:14.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:48:14.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:48:14.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:48:15.014 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:15.003+0000 7f5d9d103a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:15.016 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:15.007+0000 7f5d9d103a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:15.017 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:15.007+0000 7f5d9d103a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:15.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:48:15.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:48:15.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:48:15.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:48:15.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:48:15.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:15.155 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:48:15.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:48:15.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:15.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:48:15.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:48:15.381 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:15.371+0000 7f5d9d103a40 -1 Falling back to public interface 2026-02-22T20:48:16.005 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:15.995+0000 7f5d9d103a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:48:16.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:48:16.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:16.313 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:48:16.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:48:16.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:16.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:48:16.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:48:17.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:48:17.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:17.481 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:48:17.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:48:17.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:17.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:48:17.639 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/788881880,v1:127.0.0.1:6811/788881880] [v2:127.0.0.1:6812/788881880,v1:127.0.0.1:6813/788881880] exists,up 45c02d4d-58fb-43c7-821c-b2d7e1a47d10 2026-02-22T20:48:17.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:48:17.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:48:17.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:48:17.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:729: TEST_repair_stats: create_pool testpool 1 1 2026-02-22T20:48:17.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 2026-02-22T20:48:17.845 INFO:tasks.workunit.client.0.trial150.stderr:pool 'testpool' created 2026-02-22T20:48:17.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:48:18.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:730: TEST_repair_stats: ceph osd pool set testpool size 2 2026-02-22T20:48:19.062 INFO:tasks.workunit.client.0.trial150.stderr:set pool 1 size to 2 2026-02-22T20:48:19.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:731: TEST_repair_stats: wait_for_clean 2026-02-22T20:48:19.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:48:19.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:48:19.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:48:19.078 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:48:19.079 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:48:19.079 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:48:19.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:48:19.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:48:19.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:48:19.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:48:19.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:48:19.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:48:19.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:48:19.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:48:19.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:48:19.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:48:19.357 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:48:19.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:48:19.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:19.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:48:19.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:48:19.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:48:19.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:48:19.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:19.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:48:19.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672962 2026-02-22T20:48:19.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672962 2026-02-22T20:48:19.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672962' 2026-02-22T20:48:19.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:19.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:48:19.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:19.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:48:19.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:48:19.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:19.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:48:19.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:48:19.508 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:48:19.508 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:19.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:48:19.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:48:20.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:48:20.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:20.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T20:48:20.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:20.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672962 2026-02-22T20:48:20.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:20.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:48:20.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672962 2026-02-22T20:48:20.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:20.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672962 2026-02-22T20:48:20.837 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672962 2026-02-22T20:48:20.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672962' 2026-02-22T20:48:20.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:48:21.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672962 -lt 42949672962 2026-02-22T20:48:21.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:48:21.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:48:21.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:48:21.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:48:21.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:48:21.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:48:21.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:48:21.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:48:21.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:48:21.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:48:21.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:48:21.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:48:21.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:48:21.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:48:21.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:48:21.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:48:21.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:48:21.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:48:21.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:734: TEST_repair_stats: local payload=ABCDEF 2026-02-22T20:48:21.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:735: TEST_repair_stats: echo ABCDEF 2026-02-22T20:48:21.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: seq 1 30 2026-02-22T20:48:21.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:21.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:21.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:22.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:22.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:22.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:22.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:22.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:22.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:22.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:22.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:22.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:22.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:22.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:22.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:22.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:22.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:22.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:22.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:22.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:22.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:22.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:22.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:22.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:736: TEST_repair_stats: for i in $(seq 1 $OBJS) 2026-02-22T20:48:22.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:738: TEST_repair_stats: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:22.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:743: TEST_repair_stats: get_not_primary testpool obj1 2026-02-22T20:48:22.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:48:22.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-22T20:48:22.285 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2026-02-22T20:48:22.285 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:48:22.285 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:48:22.286 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-22T20:48:22.286 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:48:22.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:48:22.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2026-02-22T20:48:22.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:48:22.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:743: TEST_repair_stats: local other=0 2026-02-22T20:48:22.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats: get_pg testpool obj1 2026-02-22T20:48:22.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-22T20:48:22.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2026-02-22T20:48:22.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2026-02-22T20:48:22.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:48:22.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:744: TEST_repair_stats: local pgid=1.0 2026-02-22T20:48:22.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats: get_primary testpool obj1 2026-02-22T20:48:22.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:48:22.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:48:22.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:48:22.771 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-22T20:48:22.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:745: TEST_repair_stats: local primary=1 2026-02-22T20:48:22.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:747: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:48:22.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:48:22.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:48:22.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:48:22.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:48:22.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:48:23.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:48:23.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:748: TEST_repair_stats: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:48:23.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:48:23.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:48:23.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:48:23.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:48:23.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:48:23.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:48:23.552 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: seq 1 20 2026-02-22T20:48:23.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:23.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 1 % 2 2026-02-22T20:48:23.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-22T20:48:23.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2026-02-22T20:48:23.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:23.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:23.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:48:23.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:23.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:23.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2026-02-22T20:48:23.752 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-22T20:48:23.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:23.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 2 % 2 2026-02-22T20:48:23.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-22T20:48:23.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2026-02-22T20:48:23.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:23.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:23.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:48:23.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:23.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:23.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2026-02-22T20:48:24.027 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:104778fc:::obj2:head# 2026-02-22T20:48:24.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:24.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 3 % 2 2026-02-22T20:48:24.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-22T20:48:24.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2026-02-22T20:48:24.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:24.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:24.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:48:24.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:24.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:24.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2026-02-22T20:48:24.314 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:8dd16f86:::obj3:head# 2026-02-22T20:48:24.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:24.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 4 % 2 2026-02-22T20:48:24.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-22T20:48:24.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2026-02-22T20:48:24.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:24.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:24.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:48:24.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:24.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:24.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2026-02-22T20:48:24.563 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:0ee9ae15:::obj4:head# 2026-02-22T20:48:24.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:24.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 5 % 2 2026-02-22T20:48:24.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-22T20:48:24.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2026-02-22T20:48:24.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:24.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:24.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:48:24.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:24.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:24.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2026-02-22T20:48:24.870 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:c52c9666:::obj5:head# 2026-02-22T20:48:24.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:24.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 6 % 2 2026-02-22T20:48:24.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-22T20:48:24.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2026-02-22T20:48:24.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:24.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:24.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:48:24.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:24.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:24.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2026-02-22T20:48:25.157 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:233a42c1:::obj6:head# 2026-02-22T20:48:25.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:25.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 7 % 2 2026-02-22T20:48:25.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-22T20:48:25.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2026-02-22T20:48:25.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:25.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:25.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:48:25.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:25.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:25.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2026-02-22T20:48:25.442 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:9a09113a:::obj7:head# 2026-02-22T20:48:25.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:25.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 8 % 2 2026-02-22T20:48:25.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-22T20:48:25.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2026-02-22T20:48:25.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:25.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:25.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:48:25.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:25.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:25.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2026-02-22T20:48:25.714 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:a557efb1:::obj8:head# 2026-02-22T20:48:25.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:25.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 9 % 2 2026-02-22T20:48:25.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-22T20:48:25.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2026-02-22T20:48:25.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:25.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:25.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:48:25.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:25.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:25.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2026-02-22T20:48:25.970 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:fe351e27:::obj9:head# 2026-02-22T20:48:26.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:26.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 10 % 2 2026-02-22T20:48:26.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-22T20:48:26.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2026-02-22T20:48:26.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:26.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:26.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:48:26.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:26.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:26.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2026-02-22T20:48:26.256 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:d1337354:::obj10:head# 2026-02-22T20:48:26.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:26.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 11 % 2 2026-02-22T20:48:26.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-22T20:48:26.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2026-02-22T20:48:26.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:26.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:26.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:48:26.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:26.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:26.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2026-02-22T20:48:26.534 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:eeae2a94:::obj11:head# 2026-02-22T20:48:26.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:26.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 12 % 2 2026-02-22T20:48:26.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-22T20:48:26.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2026-02-22T20:48:26.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:26.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:26.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:48:26.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:26.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:26.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2026-02-22T20:48:26.822 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:af5f95cb:::obj12:head# 2026-02-22T20:48:26.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:26.924 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 13 % 2 2026-02-22T20:48:26.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-22T20:48:26.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2026-02-22T20:48:26.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:26.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:26.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:48:26.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:26.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:26.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2026-02-22T20:48:27.098 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:a61ad63e:::obj13:head# 2026-02-22T20:48:27.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:27.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 14 % 2 2026-02-22T20:48:27.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-22T20:48:27.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2026-02-22T20:48:27.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:27.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:27.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:48:27.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:27.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:27.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2026-02-22T20:48:27.379 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:e0a44829:::obj14:head# 2026-02-22T20:48:27.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:27.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 15 % 2 2026-02-22T20:48:27.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-22T20:48:27.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2026-02-22T20:48:27.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:27.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:27.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:48:27.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:27.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:27.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2026-02-22T20:48:27.654 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:ab946124:::obj15:head# 2026-02-22T20:48:27.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:27.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 16 % 2 2026-02-22T20:48:27.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-22T20:48:27.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2026-02-22T20:48:27.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:27.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:27.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:48:27.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:27.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:27.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2026-02-22T20:48:27.950 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:461f8b5e:::obj16:head# 2026-02-22T20:48:28.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:28.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 17 % 2 2026-02-22T20:48:28.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-22T20:48:28.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2026-02-22T20:48:28.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:28.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:28.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:48:28.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:28.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:28.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2026-02-22T20:48:28.215 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:24fc6e92:::obj17:head# 2026-02-22T20:48:28.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:28.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 18 % 2 2026-02-22T20:48:28.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-22T20:48:28.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2026-02-22T20:48:28.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:28.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:28.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:48:28.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:28.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:28.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2026-02-22T20:48:28.505 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:b9836a99:::obj18:head# 2026-02-22T20:48:28.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:28.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 19 % 2 2026-02-22T20:48:28.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=1 2026-02-22T20:48:28.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2026-02-22T20:48:28.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:28.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:28.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:48:28.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:28.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:28.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2026-02-22T20:48:28.782 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:00b31b6c:::obj19:head# 2026-02-22T20:48:28.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:749: TEST_repair_stats: for i in $(seq 1 $REPAIRS) 2026-02-22T20:48:28.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: expr 20 % 2 2026-02-22T20:48:28.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:752: TEST_repair_stats: OSD=0 2026-02-22T20:48:28.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:753: TEST_repair_stats: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2026-02-22T20:48:28.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:48:28.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:48:28.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:48:28.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:48:28.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:28.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2026-02-22T20:48:29.014 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:24b5611b:::obj20:head# 2026-02-22T20:48:29.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:755: TEST_repair_stats: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:29.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:48:29.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:48:29.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:48:29.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:48:29.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:29.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:48:29.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:48:29.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:48:29.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:48:29.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:48:29.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:48:29.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:48:29.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:48:29.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:48:29.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:48:29.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:48:29.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:48:29.113 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:48:29.113 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:29.113 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:29.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:29.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:29.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:48:29.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:48:29.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:48:29.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:48:29.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:48:29.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:48:29.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:48:29.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:48:29.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:48:29.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:48:29.114 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:48:29.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:48:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:29.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:48:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:48:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:48:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:48:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:48:29.137 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:29.127+0000 7f9dc47eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:29.138 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:29.127+0000 7f9dc47eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:29.139 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:29.131+0000 7f9dc47eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:29.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:48:29.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:48:29.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:48:29.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:48:29.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:48:29.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:29.275 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:48:29.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:48:29.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:29.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:48:29.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:48:29.524 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:29.515+0000 7f9dc47eca40 -1 Falling back to public interface 2026-02-22T20:48:30.198 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:30.187+0000 7f9dc47eca40 -1 osd.1 18 log_to_monitors true 2026-02-22T20:48:30.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:48:30.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:30.435 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:48:30.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:48:30.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:30.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:48:30.590 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [10,18) [v2:127.0.0.1:6802/1269179151,v1:127.0.0.1:6803/1269179151] [v2:127.0.0.1:6804/1269179151,v1:127.0.0.1:6805/1269179151] exists,up 45c02d4d-58fb-43c7-821c-b2d7e1a47d10 2026-02-22T20:48:30.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:48:30.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:48:30.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:48:30.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:756: TEST_repair_stats: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:30.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:48:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:48:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:48:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:48:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:48:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:48:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:48:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:48:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:48:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:48:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:48:30.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:48:30.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:48:30.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:48:30.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:48:30.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:48:30.595 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:48:30.595 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:30.595 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:30.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:30.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:30.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:48:30.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:48:30.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:48:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:48:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:48:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:48:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:48:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:48:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:48:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:48:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:48:30.597 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:48:30.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:30.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:48:30.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:48:30.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:48:30.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:48:30.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:48:30.619 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:30.611+0000 7fa2cd3a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:30.621 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:30.611+0000 7fa2cd3a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:30.622 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:30.611+0000 7fa2cd3a8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:30.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:48:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:48:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:48:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:48:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:48:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:30.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:48:30.758 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:48:30.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:30.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:48:30.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:48:31.032 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:31.023+0000 7fa2cd3a8a40 -1 Falling back to public interface 2026-02-22T20:48:31.602 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:31.591+0000 7fa2cd3a8a40 -1 osd.0 16 log_to_monitors true 2026-02-22T20:48:31.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:48:31.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:31.915 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:48:31.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:48:31.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:31.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:48:32.073 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 24 up_thru 0 down_at 17 last_clean_interval [5,16) [v2:127.0.0.1:6810/3562188820,v1:127.0.0.1:6811/3562188820] [v2:127.0.0.1:6812/3562188820,v1:127.0.0.1:6813/3562188820] exists,up 55b3afd0-21a2-4bc8-a7e8-e9f19f0e44d1 2026-02-22T20:48:32.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:48:32.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:48:32.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:48:32.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:757: TEST_repair_stats: wait_for_clean 2026-02-22T20:48:32.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:48:32.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:48:32.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:48:32.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:48:32.075 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:48:32.076 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:48:32.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:48:32.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:48:32.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:48:32.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:48:32.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:48:32.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:48:32.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:48:32.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:48:32.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:48:32.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:48:32.356 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:48:32.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:48:32.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:32.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:48:32.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-22T20:48:32.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-22T20:48:32.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-22T20:48:32.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:32.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:48:32.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313218 2026-02-22T20:48:32.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313218 2026-02-22T20:48:32.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-90194313218' 2026-02-22T20:48:32.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:32.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-22T20:48:32.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:32.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:48:32.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-22T20:48:32.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:32.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-22T20:48:32.506 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 103079215106 2026-02-22T20:48:32.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-22T20:48:32.507 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:32.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 103079215106 2026-02-22T20:48:32.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:48:33.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:48:33.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:33.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215106 2026-02-22T20:48:33.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:33.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-90194313218 2026-02-22T20:48:33.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:33.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:48:33.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-90194313218 2026-02-22T20:48:33.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:33.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313218 2026-02-22T20:48:33.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 90194313218' 2026-02-22T20:48:33.842 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 90194313218 2026-02-22T20:48:33.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:48:34.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313218 -lt 90194313218 2026-02-22T20:48:34.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:48:34.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:48:34.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:48:34.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:48:34.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:48:34.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:48:34.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:48:34.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:48:34.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:48:34.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:48:34.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:48:34.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:48:34.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:48:34.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:48:34.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:48:34.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:48:34.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:48:34.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:48:34.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:759: TEST_repair_stats: repair 1.0 2026-02-22T20:48:34.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-22T20:48:34.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-22T20:48:34.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:48:34.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:48:34.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:48:34.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:48:34.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:48:17.837070+0000 2026-02-22T20:48:34.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-22T20:48:34.883 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0 on osd.1 to repair 2026-02-22T20:48:34.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-22T20:48:17.837070+0000 2026-02-22T20:48:34.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:48:34.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:48:17.837070+0000 2026-02-22T20:48:34.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:48:34.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:48:34.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:48:34.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:48:34.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:48:34.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:48:34.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:48:34.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:48:35.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:48:17.837070+0000 '>' 2026-02-22T20:48:17.837070+0000 2026-02-22T20:48:35.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:48:36.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:48:36.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:48:36.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:48:36.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:48:36.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:48:36.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:48:36.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:48:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:48:17.837070+0000 '>' 2026-02-22T20:48:17.837070+0000 2026-02-22T20:48:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:48:37.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:48:37.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:48:37.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:48:37.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:48:37.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:48:37.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:48:37.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:48:37.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:48:35.082991+0000 '>' 2026-02-22T20:48:17.837070+0000 2026-02-22T20:48:37.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:48:37.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:760: TEST_repair_stats: wait_for_clean 2026-02-22T20:48:37.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:48:37.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:48:37.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:48:37.389 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:48:37.390 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:48:37.390 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:48:37.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:48:37.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:48:37.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:48:37.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:48:37.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:48:37.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:48:37.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:48:37.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:48:37.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:48:37.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:48:37.663 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:48:37.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:48:37.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:37.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:48:37.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215108 2026-02-22T20:48:37.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215108 2026-02-22T20:48:37.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108' 2026-02-22T20:48:37.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:37.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:48:37.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313220 2026-02-22T20:48:37.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313220 2026-02-22T20:48:37.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108 1-90194313220' 2026-02-22T20:48:37.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:37.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215108 2026-02-22T20:48:37.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:37.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:48:37.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215108 2026-02-22T20:48:37.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:37.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215108 2026-02-22T20:48:37.815 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 103079215108 2026-02-22T20:48:37.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2026-02-22T20:48:37.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:37.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215108 2026-02-22T20:48:37.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:48:38.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:48:38.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:39.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215108 -lt 103079215108 2026-02-22T20:48:39.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:39.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-90194313220 2026-02-22T20:48:39.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:39.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:48:39.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-90194313220 2026-02-22T20:48:39.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:39.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313220 2026-02-22T20:48:39.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 90194313220' 2026-02-22T20:48:39.140 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 90194313220 2026-02-22T20:48:39.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:48:39.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313220 -lt 90194313220 2026-02-22T20:48:39.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:48:39.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:48:39.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:48:39.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:48:39.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:48:39.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:48:39.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:48:39.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:48:39.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:48:39.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:48:39.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:48:39.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:48:39.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:48:39.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:48:39.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:48:39.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:48:39.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:48:39.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:48:39.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:761: TEST_repair_stats: ceph pg dump pgs 2026-02-22T20:48:39.984 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:48:39.984 INFO:tasks.workunit.client.0.trial150.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2026-02-22T20:48:35.099675+0000 16'30 25:130 [1,0] 1 [1,0] 1 16'30 2026-02-22T20:48:35.082991+0000 16'30 2026-02-22T20:48:35.082991+0000 0 1 periodic scrub scheduled @ 2026-02-23T20:48:35.082991+0000 20 0 2026-02-22T20:48:39.984 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:48:39.984 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:48:39.984 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:48:39.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:762: TEST_repair_stats: flush_pg_stats 2026-02-22T20:48:39.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:48:39.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:48:40.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:48:40.165 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:48:40.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:48:40.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:40.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:48:40.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215109 2026-02-22T20:48:40.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215109 2026-02-22T20:48:40.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215109' 2026-02-22T20:48:40.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:40.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:48:40.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=90194313222 2026-02-22T20:48:40.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 90194313222 2026-02-22T20:48:40.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215109 1-90194313222' 2026-02-22T20:48:40.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:40.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215109 2026-02-22T20:48:40.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:40.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:48:40.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215109 2026-02-22T20:48:40.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:40.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215109 2026-02-22T20:48:40.318 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 103079215109 2026-02-22T20:48:40.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215109' 2026-02-22T20:48:40.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:40.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215108 -lt 103079215109 2026-02-22T20:48:40.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:48:41.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:48:41.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:41.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215109 -lt 103079215109 2026-02-22T20:48:41.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:41.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-90194313222 2026-02-22T20:48:41.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:41.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:48:41.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-90194313222 2026-02-22T20:48:41.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:41.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=90194313222 2026-02-22T20:48:41.647 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 90194313222 2026-02-22T20:48:41.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 90194313222' 2026-02-22T20:48:41.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:48:41.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 90194313222 -lt 90194313222 2026-02-22T20:48:41.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats: ceph pg 1.0 query 2026-02-22T20:48:41.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:765: TEST_repair_stats: jq .info.stats.stat_sum 2026-02-22T20:48:41.885 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:48:41.885 INFO:tasks.workunit.client.0.trial150.stdout: "num_bytes": 210, 2026-02-22T20:48:41.885 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects": 30, 2026-02-22T20:48:41.885 INFO:tasks.workunit.client.0.trial150.stdout: "num_object_clones": 0, 2026-02-22T20:48:41.885 INFO:tasks.workunit.client.0.trial150.stdout: "num_object_copies": 60, 2026-02-22T20:48:41.885 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T20:48:41.885 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_missing": 0, 2026-02-22T20:48:41.885 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_degraded": 0, 2026-02-22T20:48:41.885 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_misplaced": 0, 2026-02-22T20:48:41.885 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_unfound": 0, 2026-02-22T20:48:41.885 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_dirty": 30, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_whiteouts": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_read": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_read_kb": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_write": 30, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_write_kb": 30, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_scrub_errors": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_deep_scrub_errors": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_recovered": 20, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_bytes_recovered": 140, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_keys_recovered": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_omap": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_flush": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_flush_kb": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_evict": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_evict_kb": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_promote": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_flush_mode_high": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_flush_mode_low": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_evict_mode_some": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_evict_mode_full": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_pinned": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_legacy_snapsets": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_large_omap_objects": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_manifest": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_omap_bytes": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_omap_keys": 0, 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_repaired": 20 2026-02-22T20:48:41.886 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:48:41.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats: ceph pg 1.0 query 2026-02-22T20:48:41.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats: jq .info.stats.stat_sum.num_objects_repaired 2026-02-22T20:48:41.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:766: TEST_repair_stats: COUNT=20 2026-02-22T20:48:41.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:767: TEST_repair_stats: test 20 = 20 2026-02-22T20:48:41.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:769: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-22T20:48:41.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:769: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2026-02-22T20:48:42.104 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:48:42.113 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:48:42.114 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:48:42.114 INFO:tasks.workunit.client.0.trial150.stdout: "up_from": 21, 2026-02-22T20:48:42.114 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 90194313222, 2026-02-22T20:48:42.114 INFO:tasks.workunit.client.0.trial150.stdout: "num_pgs": 1, 2026-02-22T20:48:42.114 INFO:tasks.workunit.client.0.trial150.stdout: "num_osds": 1, 2026-02-22T20:48:42.114 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_osds": 1, 2026-02-22T20:48:42.115 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_omap_osds": 1, 2026-02-22T20:48:42.115 INFO:tasks.workunit.client.0.trial150.stdout: "kb": 104857600, 2026-02-22T20:48:42.115 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used": 28568, 2026-02-22T20:48:42.115 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_data": 328, 2026-02-22T20:48:42.115 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_omap": 7, 2026-02-22T20:48:42.115 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_meta": 28216, 2026-02-22T20:48:42.116 INFO:tasks.workunit.client.0.trial150.stdout: "kb_avail": 104829032, 2026-02-22T20:48:42.116 INFO:tasks.workunit.client.0.trial150.stdout: "statfs": { 2026-02-22T20:48:42.116 INFO:tasks.workunit.client.0.trial150.stdout: "total": 107374182400, 2026-02-22T20:48:42.116 INFO:tasks.workunit.client.0.trial150.stdout: "available": 107344928768, 2026-02-22T20:48:42.116 INFO:tasks.workunit.client.0.trial150.stdout: "internally_reserved": 0, 2026-02-22T20:48:42.116 INFO:tasks.workunit.client.0.trial150.stdout: "allocated": 335872, 2026-02-22T20:48:42.116 INFO:tasks.workunit.client.0.trial150.stdout: "data_stored": 55763, 2026-02-22T20:48:42.117 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed": 0, 2026-02-22T20:48:42.117 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_allocated": 0, 2026-02-22T20:48:42.117 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_original": 0, 2026-02-22T20:48:42.117 INFO:tasks.workunit.client.0.trial150.stdout: "omap_allocated": 7653, 2026-02-22T20:48:42.117 INFO:tasks.workunit.client.0.trial150.stdout: "internal_metadata": 28893723 2026-02-22T20:48:42.117 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:48:42.117 INFO:tasks.workunit.client.0.trial150.stdout: "hb_peers": [ 2026-02-22T20:48:42.118 INFO:tasks.workunit.client.0.trial150.stdout: 0 2026-02-22T20:48:42.118 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:48:42.118 INFO:tasks.workunit.client.0.trial150.stdout: "snap_trim_queue_len": 0, 2026-02-22T20:48:42.118 INFO:tasks.workunit.client.0.trial150.stdout: "num_snap_trimming": 0, 2026-02-22T20:48:42.118 INFO:tasks.workunit.client.0.trial150.stdout: "num_shards_repaired": 10, 2026-02-22T20:48:42.118 INFO:tasks.workunit.client.0.trial150.stdout: "op_queue_age_hist": { 2026-02-22T20:48:42.118 INFO:tasks.workunit.client.0.trial150.stdout: "histogram": [], 2026-02-22T20:48:42.119 INFO:tasks.workunit.client.0.trial150.stdout: "upper_bound": 1 2026-02-22T20:48:42.119 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:48:42.119 INFO:tasks.workunit.client.0.trial150.stdout: "perf_stat": { 2026-02-22T20:48:42.119 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ms": 0, 2026-02-22T20:48:42.119 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ms": 0, 2026-02-22T20:48:42.119 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ns": 0, 2026-02-22T20:48:42.119 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ns": 0 2026-02-22T20:48:42.119 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:48:42.120 INFO:tasks.workunit.client.0.trial150.stdout: "alerts": [] 2026-02-22T20:48:42.120 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:48:42.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-22T20:48:42.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2026-02-22T20:48:42.267 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:48:42.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:770: TEST_repair_stats: COUNT=10 2026-02-22T20:48:42.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats: expr 20 / 2 2026-02-22T20:48:42.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:771: TEST_repair_stats: test 10 = 10 2026-02-22T20:48:42.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-22T20:48:42.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:773: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2026-02-22T20:48:42.425 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:48:42.435 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:48:42.435 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:48:42.435 INFO:tasks.workunit.client.0.trial150.stdout: "up_from": 24, 2026-02-22T20:48:42.435 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 103079215109, 2026-02-22T20:48:42.435 INFO:tasks.workunit.client.0.trial150.stdout: "num_pgs": 1, 2026-02-22T20:48:42.435 INFO:tasks.workunit.client.0.trial150.stdout: "num_osds": 1, 2026-02-22T20:48:42.435 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_osds": 1, 2026-02-22T20:48:42.435 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_omap_osds": 1, 2026-02-22T20:48:42.435 INFO:tasks.workunit.client.0.trial150.stdout: "kb": 104857600, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used": 28568, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_data": 328, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_omap": 5, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_meta": 28218, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "kb_avail": 104829032, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "statfs": { 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "total": 107374182400, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "available": 107344928768, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "internally_reserved": 0, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "allocated": 335872, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "data_stored": 55763, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed": 0, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_allocated": 0, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_original": 0, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "omap_allocated": 5702, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "internal_metadata": 28895674 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "hb_peers": [ 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "snap_trim_queue_len": 0, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "num_snap_trimming": 0, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "num_shards_repaired": 10, 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "op_queue_age_hist": { 2026-02-22T20:48:42.436 INFO:tasks.workunit.client.0.trial150.stdout: "histogram": [], 2026-02-22T20:48:42.437 INFO:tasks.workunit.client.0.trial150.stdout: "upper_bound": 1 2026-02-22T20:48:42.437 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:48:42.437 INFO:tasks.workunit.client.0.trial150.stdout: "perf_stat": { 2026-02-22T20:48:42.437 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ms": 0, 2026-02-22T20:48:42.437 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ms": 0, 2026-02-22T20:48:42.437 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ns": 0, 2026-02-22T20:48:42.437 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ns": 0 2026-02-22T20:48:42.437 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:48:42.437 INFO:tasks.workunit.client.0.trial150.stdout: "alerts": [] 2026-02-22T20:48:42.437 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:48:42.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-22T20:48:42.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2026-02-22T20:48:42.584 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:48:42.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:774: TEST_repair_stats: COUNT=10 2026-02-22T20:48:42.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:775: TEST_repair_stats: expr 20 / 2 2026-02-22T20:48:42.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:775: TEST_repair_stats: test 10 = 10 2026-02-22T20:48:42.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-22T20:48:42.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:777: TEST_repair_stats: jq .pg_map.osd_stats_sum 2026-02-22T20:48:42.742 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "up_from": 0, 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 0, 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "num_pgs": 2, 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "num_osds": 2, 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_osds": 2, 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_omap_osds": 2, 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "kb": 209715200, 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used": 57136, 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_data": 656, 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_omap": 13, 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_meta": 56434, 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "kb_avail": 209658064, 2026-02-22T20:48:42.753 INFO:tasks.workunit.client.0.trial150.stdout: "statfs": { 2026-02-22T20:48:42.754 INFO:tasks.workunit.client.0.trial150.stdout: "total": 214748364800, 2026-02-22T20:48:42.754 INFO:tasks.workunit.client.0.trial150.stdout: "available": 214689857536, 2026-02-22T20:48:42.754 INFO:tasks.workunit.client.0.trial150.stdout: "internally_reserved": 0, 2026-02-22T20:48:42.754 INFO:tasks.workunit.client.0.trial150.stdout: "allocated": 671744, 2026-02-22T20:48:42.754 INFO:tasks.workunit.client.0.trial150.stdout: "data_stored": 111526, 2026-02-22T20:48:42.754 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed": 0, 2026-02-22T20:48:42.754 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_allocated": 0, 2026-02-22T20:48:42.754 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_original": 0, 2026-02-22T20:48:42.754 INFO:tasks.workunit.client.0.trial150.stdout: "omap_allocated": 13355, 2026-02-22T20:48:42.754 INFO:tasks.workunit.client.0.trial150.stdout: "internal_metadata": 57789397 2026-02-22T20:48:42.754 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:48:42.754 INFO:tasks.workunit.client.0.trial150.stdout: "hb_peers": [], 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: "snap_trim_queue_len": 0, 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: "num_snap_trimming": 0, 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: "num_shards_repaired": 20, 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: "op_queue_age_hist": { 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: "histogram": [], 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: "upper_bound": 1 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: "perf_stat": { 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ms": 0, 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ms": 0, 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ns": 0, 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ns": 0 2026-02-22T20:48:42.755 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:48:42.756 INFO:tasks.workunit.client.0.trial150.stdout: "alerts": [], 2026-02-22T20:48:42.756 INFO:tasks.workunit.client.0.trial150.stdout: "network_ping_times": [] 2026-02-22T20:48:42.756 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:48:42.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats: ceph pg dump --format=json-pretty 2026-02-22T20:48:42.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats: jq .pg_map.osd_stats_sum.num_shards_repaired 2026-02-22T20:48:42.905 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:48:42.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:778: TEST_repair_stats: COUNT=20 2026-02-22T20:48:42.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:779: TEST_repair_stats: test 20 = 20 2026-02-22T20:48:42.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:48:42.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:48:42.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:48:42.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:48:42.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:48:42.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:48:42.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:48:42.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:48:42.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:48:43.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:48:43.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:48:43.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:48:43.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:48:43.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:48:43.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:48:43.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:48:43.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:48:43.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:48:43.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:48:43.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:48:43.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:48:43.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:48:43.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:48:43.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:48:43.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:43.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:43.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:48:43.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:48:43.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:48:43.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:48:43.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:48:43.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:48:43.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:48:43.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:48:43.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:48:43.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:48:43.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:48:43.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:48:43.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:48:43.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:48:43.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:48:43.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:48:43.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:48:43.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:48:43.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:48:43.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:48:43.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:48:43.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:48:43.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:48:43.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:48:43.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:48:43.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:48:43.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:48:43.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:48:43.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:48:43.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:48:43.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:43.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:43.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:48:43.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:48:43.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:48:43.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:48:43.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:48:43.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:43.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:43.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:48:43.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_repair_stats_ec td/osd-scrub-repair 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:783: TEST_repair_stats_ec: local dir=td/osd-scrub-repair 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:784: TEST_repair_stats_ec: local poolname=testpool 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:785: TEST_repair_stats_ec: local OSDS=3 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:786: TEST_repair_stats_ec: local OBJS=30 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:788: TEST_repair_stats_ec: local REPAIRS=26 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:789: TEST_repair_stats_ec: local allow_overwrites=false 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:792: TEST_repair_stats_ec: run_mon td/osd-scrub-repair a 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:48:43.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:48:43.090 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:48:43.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:48:43.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:48:43.091 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:48:43.091 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:43.092 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:43.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:43.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:48:43.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:48:43.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:48:43.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:48:43.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:48:43.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:48:43.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:48:43.123 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:48:43.124 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:48:43.124 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:48:43.124 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:48:43.124 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:43.124 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:43.124 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:48:43.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:48:43.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:48:43.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:48:43.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:48:43.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:48:43.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:48:43.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:48:43.180 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:48:43.180 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:48:43.180 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:48:43.180 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:48:43.180 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:43.180 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:43.181 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:48:43.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:48:43.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:48:43.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:793: TEST_repair_stats_ec: run_mgr td/osd-scrub-repair x 2026-02-22T20:48:43.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:48:43.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:48:43.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:48:43.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:48:43.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:48:43.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:48:43.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:48:43.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:48:43.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:48:43.332 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:48:43.332 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:43.332 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:43.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:43.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:48:43.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:48:43.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:795: TEST_repair_stats_ec: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:48:43.354 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: expr 3 - 1 2026-02-22T20:48:43.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: seq 0 2 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:797: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:48:43.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:48:43.358 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:48:43.358 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:43.358 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:43.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:43.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:43.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:48:43.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:48:43.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:48:43.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:48:43.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:48:43.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:48:43.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:48:43.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:48:43.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:48:43.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:48:43.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:48:43.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:48:43.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:48:43.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e3d32c08-79e4-422e-aa50-e0947ef8188c 2026-02-22T20:48:43.361 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 e3d32c08-79e4-422e-aa50-e0947ef8188c 2026-02-22T20:48:43.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e3d32c08-79e4-422e-aa50-e0947ef8188c' 2026-02-22T20:48:43.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:48:43.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCra5tpjWfCFRAA6G0QtoUHCDCelUvkmtj2yg== 2026-02-22T20:48:43.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCra5tpjWfCFRAA6G0QtoUHCDCelUvkmtj2yg=="}' 2026-02-22T20:48:43.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e3d32c08-79e4-422e-aa50-e0947ef8188c -i td/osd-scrub-repair/0/new.json 2026-02-22T20:48:43.461 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:48:43.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:48:43.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCra5tpjWfCFRAA6G0QtoUHCDCelUvkmtj2yg== --osd-uuid e3d32c08-79e4-422e-aa50-e0947ef8188c 2026-02-22T20:48:43.486 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:43.475+0000 7f1bf1869a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:43.487 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:43.479+0000 7f1bf1869a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:43.488 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:43.479+0000 7f1bf1869a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:43.489 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:43.479+0000 7f1bf1869a40 -1 bdev(0x563adf694c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:48:43.489 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:43.479+0000 7f1bf1869a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:48:44.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:48:44.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:48:44.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:48:44.235 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:48:44.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:48:44.437 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:48:44.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:48:44.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:44.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:48:44.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:48:44.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:48:44.462 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:44.455+0000 7fae7b53ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:44.464 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:44.455+0000 7fae7b53ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:44.465 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:44.455+0000 7fae7b53ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:44.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:48:44.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:48:44.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:48:44.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:48:44.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:48:44.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:44.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:48:44.602 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:48:44.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:44.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:48:44.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:48:44.848 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:44.839+0000 7fae7b53ba40 -1 Falling back to public interface 2026-02-22T20:48:45.472 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:45.463+0000 7fae7b53ba40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:48:45.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:48:45.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:45.757 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:48:45.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:48:45.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:45.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:48:45.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:48:46.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:48:46.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:46.916 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:48:46.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:48:46.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:46.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:48:47.077 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4247962833,v1:127.0.0.1:6803/4247962833] [v2:127.0.0.1:6804/4247962833,v1:127.0.0.1:6805/4247962833] exists,up e3d32c08-79e4-422e-aa50-e0947ef8188c 2026-02-22T20:48:47.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:48:47.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:797: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:48:47.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:48:47.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:48:47.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:48:47.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:48:47.080 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:48:47.080 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:47.080 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:47.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:48:47.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:48:47.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:48:47.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7af23ca5-ba78-49ad-8390-906c5e1d1d71 2026-02-22T20:48:47.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 7af23ca5-ba78-49ad-8390-906c5e1d1d71' 2026-02-22T20:48:47.083 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 7af23ca5-ba78-49ad-8390-906c5e1d1d71 2026-02-22T20:48:47.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:48:47.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCva5tpab9CBRAABUUPYue5R47Q7yDK81cgXQ== 2026-02-22T20:48:47.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCva5tpab9CBRAABUUPYue5R47Q7yDK81cgXQ=="}' 2026-02-22T20:48:47.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7af23ca5-ba78-49ad-8390-906c5e1d1d71 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:48:47.257 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:48:47.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:48:47.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCva5tpab9CBRAABUUPYue5R47Q7yDK81cgXQ== --osd-uuid 7af23ca5-ba78-49ad-8390-906c5e1d1d71 2026-02-22T20:48:47.295 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:47.287+0000 7f003d685a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:47.296 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:47.287+0000 7f003d685a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:47.298 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:47.287+0000 7f003d685a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:47.298 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:47.287+0000 7f003d685a40 -1 bdev(0x55c324414c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:48:47.299 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:47.291+0000 7f003d685a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:48:47.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:48:47.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:48:47.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:48:47.939 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:48:47.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:48:48.141 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:48:48.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:48:48.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:48.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:48:48.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:48:48.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:48:48.158 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:48.147+0000 7f7380af5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:48.159 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:48.151+0000 7f7380af5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:48.161 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:48.151+0000 7f7380af5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:48.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:48:48.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:48:48.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:48:48.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:48:48.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:48:48.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:48.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:48:48.299 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:48:48.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:48.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:48:48.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:48:48.512 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:48.503+0000 7f7380af5a40 -1 Falling back to public interface 2026-02-22T20:48:49.066 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:49.055+0000 7f7380af5a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:48:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:48:49.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:49.461 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:48:49.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:48:49.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:49.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:48:49.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:48:50.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:48:50.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:50.617 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:48:50.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:48:50.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:50.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:48:50.775 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/805159050,v1:127.0.0.1:6811/805159050] [v2:127.0.0.1:6812/805159050,v1:127.0.0.1:6813/805159050] exists,up 7af23ca5-ba78-49ad-8390-906c5e1d1d71 2026-02-22T20:48:50.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:48:50.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:48:50.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:48:50.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:796: TEST_repair_stats_ec: for id in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:48:50.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:797: TEST_repair_stats_ec: run_osd td/osd-scrub-repair 2 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:50.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:48:50.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:48:50.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:48:50.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:48:50.778 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:48:50.778 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:48:50.778 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:48:50.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:50.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:48:50.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:48:50.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9e223f79-ea40-4af2-b2de-584157b31ff7 2026-02-22T20:48:50.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 9e223f79-ea40-4af2-b2de-584157b31ff7' 2026-02-22T20:48:50.780 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 9e223f79-ea40-4af2-b2de-584157b31ff7 2026-02-22T20:48:50.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:48:50.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCya5tpeljMLhAAnCRtszb59ff5ZkQZ1se17g== 2026-02-22T20:48:50.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCya5tpeljMLhAAnCRtszb59ff5ZkQZ1se17g=="}' 2026-02-22T20:48:50.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9e223f79-ea40-4af2-b2de-584157b31ff7 -i td/osd-scrub-repair/2/new.json 2026-02-22T20:48:50.954 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:48:50.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:48:50.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 --mkfs --key AQCya5tpeljMLhAAnCRtszb59ff5ZkQZ1se17g== --osd-uuid 9e223f79-ea40-4af2-b2de-584157b31ff7 2026-02-22T20:48:50.982 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:50.971+0000 7f056c0c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:50.983 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:50.975+0000 7f056c0c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:50.984 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:50.975+0000 7f056c0c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:50.985 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:50.975+0000 7f056c0c4a40 -1 bdev(0x55a4ebf82c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:48:50.985 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:50.975+0000 7f056c0c4a40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:48:51.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:48:51.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:48:51.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:48:51.595 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:48:51.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:48:51.793 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:48:51.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:48:51.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:48:51.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:48:51.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:48:51.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:48:51.809 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:51.799+0000 7fce80b54a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:51.811 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:51.803+0000 7fce80b54a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:51.812 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:51.803+0000 7fce80b54a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:48:51.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:48:51.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:48:51.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:48:51.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:48:51.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:48:51.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:51.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:48:51.954 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:48:51.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:51.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:48:52.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:48:52.204 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:52.195+0000 7fce80b54a40 -1 Falling back to public interface 2026-02-22T20:48:52.844 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:48:52.835+0000 7fce80b54a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:48:53.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:48:53.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:53.109 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:48:53.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:48:53.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:53.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:48:53.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:48:54.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:48:54.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:48:54.271 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:48:54.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:48:54.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:48:54.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:48:54.429 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1607771157,v1:127.0.0.1:6819/1607771157] [v2:127.0.0.1:6820/1607771157,v1:127.0.0.1:6821/1607771157] exists,up 9e223f79-ea40-4af2-b2de-584157b31ff7 2026-02-22T20:48:54.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:48:54.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:48:54.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:48:54.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:801: TEST_repair_stats_ec: create_ec_pool testpool false k=2 m=1 2026-02-22T20:48:54.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=testpool 2026-02-22T20:48:54.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:48:54.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-22T20:48:54.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:48:54.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2026-02-22T20:48:54.608 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:48:54.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool testpool 1 1 erasure myprofile 2026-02-22T20:48:54.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create testpool 1 1 erasure myprofile 2026-02-22T20:48:54.914 INFO:tasks.workunit.client.0.trial150.stderr:pool 'testpool' created 2026-02-22T20:48:54.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:48:55.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-22T20:48:55.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:48:55.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:48:55.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:48:55.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:48:55.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:48:55.929 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:48:55.929 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:48:55.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:48:55.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:48:55.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:48:56.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:48:56.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:48:56.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:48:56.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:48:56.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:48:56.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:48:56.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:48:56.205 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:48:56.205 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:48:56.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:48:56.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:56.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:48:56.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:48:56.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:48:56.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:48:56.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:56.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:48:56.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-22T20:48:56.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-22T20:48:56.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-22T20:48:56.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:48:56.349 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:48:56.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542146 2026-02-22T20:48:56.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542146 2026-02-22T20:48:56.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-60129542146' 2026-02-22T20:48:56.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:56.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:48:56.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:56.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:48:56.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:48:56.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:56.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:48:56.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:48:56.426 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:48:56.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:56.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:48:56.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:48:57.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:48:57.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:57.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:48:57.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:48:58.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:48:58.749 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:48:58.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-22T20:48:58.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:58.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-22T20:48:58.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:58.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:48:58.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-22T20:48:58.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:58.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-22T20:48:58.915 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672963 2026-02-22T20:48:58.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-22T20:48:58.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:48:59.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-22T20:48:59.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:48:59.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542146 2026-02-22T20:48:59.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:48:59.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:48:59.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542146 2026-02-22T20:48:59.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:48:59.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542146 2026-02-22T20:48:59.084 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542146 2026-02-22T20:48:59.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542146' 2026-02-22T20:48:59.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:48:59.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542146 -lt 60129542146 2026-02-22T20:48:59.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:48:59.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:48:59.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:48:59.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:48:59.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:48:59.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:48:59.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:48:59.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:48:59.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:48:59.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:48:59.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:48:59.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:48:59.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:48:59.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:48:59.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:48:59.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:48:59.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:48:59.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:48:59.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:48:59.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:804: TEST_repair_stats_ec: local payload=ABCDEF 2026-02-22T20:48:59.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:805: TEST_repair_stats_ec: echo ABCDEF 2026-02-22T20:48:59.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: seq 1 30 2026-02-22T20:48:59.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:48:59.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj1 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:59.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:48:59.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj2 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:59.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:48:59.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj3 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:59.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:48:59.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj4 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:59.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:48:59.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj5 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:59.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:48:59.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj6 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:59.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:48:59.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj7 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:59.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:48:59.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj8 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:48:59.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:48:59.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj9 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj10 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj11 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj12 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj13 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj14 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj15 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj16 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj17 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj18 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj19 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj20 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj21 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj22 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj23 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj24 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj25 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj26 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj27 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj28 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj29 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:806: TEST_repair_stats_ec: for i in $(seq 1 $OBJS) 2026-02-22T20:49:00.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:808: TEST_repair_stats_ec: rados --pool testpool put obj30 td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:00.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:813: TEST_repair_stats_ec: get_not_primary testpool obj1 2026-02-22T20:49:00.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=testpool 2026-02-22T20:49:00.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-22T20:49:00.555 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary testpool obj1 2026-02-22T20:49:00.555 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:49:00.555 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:49:00.556 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-22T20:49:00.556 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:49:00.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:49:00.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map testpool obj1 2026-02-22T20:49:00.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:49:00.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:813: TEST_repair_stats_ec: local other=0 2026-02-22T20:49:00.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:814: TEST_repair_stats_ec: get_pg testpool obj1 2026-02-22T20:49:00.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=testpool 2026-02-22T20:49:00.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2026-02-22T20:49:00.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map testpool obj1 2026-02-22T20:49:00.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:49:01.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:814: TEST_repair_stats_ec: local pgid=1.0 2026-02-22T20:49:01.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_repair_stats_ec: get_primary testpool obj1 2026-02-22T20:49:01.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=testpool 2026-02-22T20:49:01.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:49:01.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:49:01.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map testpool obj1 2026-02-22T20:49:01.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:815: TEST_repair_stats_ec: local primary=1 2026-02-22T20:49:01.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:817: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:49:01.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:49:01.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:49:01.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:49:01.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:49:01.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:49:01.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:49:01.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:818: TEST_repair_stats_ec: kill_daemons td/osd-scrub-repair TERM osd.1 2026-02-22T20:49:01.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:49:01.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:49:01.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:49:01.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:49:01.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:49:01.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:49:01.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: seq 1 26 2026-02-22T20:49:01.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:01.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 1 % 2 2026-02-22T20:49:01.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:01.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj1 remove 2026-02-22T20:49:01.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:01.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:01.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:01.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:01.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:01.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj1 remove 2026-02-22T20:49:02.043 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:ff7b1f36:::obj1:head# 2026-02-22T20:49:02.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:02.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 2 % 2 2026-02-22T20:49:02.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:02.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj2 remove 2026-02-22T20:49:02.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:02.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:02.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:02.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:02.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:02.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj2 remove 2026-02-22T20:49:02.342 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:104778fc:::obj2:head# 2026-02-22T20:49:02.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:02.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 3 % 2 2026-02-22T20:49:02.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:02.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj3 remove 2026-02-22T20:49:02.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:02.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:02.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:02.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:02.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:02.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj3 remove 2026-02-22T20:49:02.660 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:8dd16f86:::obj3:head# 2026-02-22T20:49:02.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:02.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 4 % 2 2026-02-22T20:49:02.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:02.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj4 remove 2026-02-22T20:49:02.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:02.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:02.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:02.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:02.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:02.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj4 remove 2026-02-22T20:49:02.950 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:0ee9ae15:::obj4:head# 2026-02-22T20:49:03.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:03.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 5 % 2 2026-02-22T20:49:03.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:03.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj5 remove 2026-02-22T20:49:03.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:03.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:03.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:03.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:03.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:03.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj5 remove 2026-02-22T20:49:03.226 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:c52c9666:::obj5:head# 2026-02-22T20:49:03.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:03.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 6 % 2 2026-02-22T20:49:03.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:03.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj6 remove 2026-02-22T20:49:03.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:03.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:03.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:03.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:03.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:03.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj6 remove 2026-02-22T20:49:03.490 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:233a42c1:::obj6:head# 2026-02-22T20:49:03.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:03.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 7 % 2 2026-02-22T20:49:03.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:03.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj7 remove 2026-02-22T20:49:03.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:03.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:03.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:03.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:03.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:03.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj7 remove 2026-02-22T20:49:03.746 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:9a09113a:::obj7:head# 2026-02-22T20:49:03.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:03.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 8 % 2 2026-02-22T20:49:03.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:03.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj8 remove 2026-02-22T20:49:03.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:03.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:03.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:03.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:03.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:03.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj8 remove 2026-02-22T20:49:04.034 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:a557efb1:::obj8:head# 2026-02-22T20:49:04.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:04.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 9 % 2 2026-02-22T20:49:04.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:04.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj9 remove 2026-02-22T20:49:04.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:04.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:04.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:04.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:04.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:04.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj9 remove 2026-02-22T20:49:04.289 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:fe351e27:::obj9:head# 2026-02-22T20:49:04.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:04.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 10 % 2 2026-02-22T20:49:04.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:04.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj10 remove 2026-02-22T20:49:04.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:04.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:04.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:04.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:04.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:04.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj10 remove 2026-02-22T20:49:04.550 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:d1337354:::obj10:head# 2026-02-22T20:49:04.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:04.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 11 % 2 2026-02-22T20:49:04.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:04.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj11 remove 2026-02-22T20:49:04.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:04.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:04.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:04.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:04.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:04.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj11 remove 2026-02-22T20:49:04.835 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:eeae2a94:::obj11:head# 2026-02-22T20:49:04.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:04.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 12 % 2 2026-02-22T20:49:04.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:04.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj12 remove 2026-02-22T20:49:04.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:04.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:04.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:04.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:04.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:04.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj12 remove 2026-02-22T20:49:05.113 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:af5f95cb:::obj12:head# 2026-02-22T20:49:05.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:05.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 13 % 2 2026-02-22T20:49:05.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:05.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj13 remove 2026-02-22T20:49:05.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:05.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:05.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:05.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:05.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:05.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj13 remove 2026-02-22T20:49:05.393 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:a61ad63e:::obj13:head# 2026-02-22T20:49:05.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:05.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 14 % 2 2026-02-22T20:49:05.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:05.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj14 remove 2026-02-22T20:49:05.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:05.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:05.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:05.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:05.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:05.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj14 remove 2026-02-22T20:49:05.678 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:e0a44829:::obj14:head# 2026-02-22T20:49:05.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:05.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 15 % 2 2026-02-22T20:49:05.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:05.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj15 remove 2026-02-22T20:49:05.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:05.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:05.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:05.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:05.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:05.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj15 remove 2026-02-22T20:49:05.962 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:ab946124:::obj15:head# 2026-02-22T20:49:06.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:06.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 16 % 2 2026-02-22T20:49:06.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:06.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj16 remove 2026-02-22T20:49:06.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:06.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:06.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:06.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:06.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:06.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj16 remove 2026-02-22T20:49:06.242 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:461f8b5e:::obj16:head# 2026-02-22T20:49:06.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:06.341 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 17 % 2 2026-02-22T20:49:06.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:06.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj17 remove 2026-02-22T20:49:06.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:06.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:06.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:06.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:06.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:06.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj17 remove 2026-02-22T20:49:06.539 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:24fc6e92:::obj17:head# 2026-02-22T20:49:06.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:06.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 18 % 2 2026-02-22T20:49:06.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:06.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj18 remove 2026-02-22T20:49:06.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:06.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:06.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:06.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:06.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:06.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj18 remove 2026-02-22T20:49:06.829 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:b9836a99:::obj18:head# 2026-02-22T20:49:06.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:06.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 19 % 2 2026-02-22T20:49:06.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:06.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj19 remove 2026-02-22T20:49:06.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:06.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:06.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:06.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:06.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:06.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj19 remove 2026-02-22T20:49:07.101 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:00b31b6c:::obj19:head# 2026-02-22T20:49:07.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:07.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 20 % 2 2026-02-22T20:49:07.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:07.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj20 remove 2026-02-22T20:49:07.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:07.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:07.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:07.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:07.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:07.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj20 remove 2026-02-22T20:49:07.402 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:24b5611b:::obj20:head# 2026-02-22T20:49:07.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:07.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 21 % 2 2026-02-22T20:49:07.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:07.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj21 remove 2026-02-22T20:49:07.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:07.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:07.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:07.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:07.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:07.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj21 remove 2026-02-22T20:49:07.682 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:1243fe1a:::obj21:head# 2026-02-22T20:49:07.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:07.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 22 % 2 2026-02-22T20:49:07.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:07.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj22 remove 2026-02-22T20:49:07.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:07.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:07.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:07.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:07.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:07.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj22 remove 2026-02-22T20:49:07.969 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:c0c49def:::obj22:head# 2026-02-22T20:49:08.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:08.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 23 % 2 2026-02-22T20:49:08.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:08.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj23 remove 2026-02-22T20:49:08.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:08.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:08.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:08.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:08.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:08.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj23 remove 2026-02-22T20:49:08.258 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:242c40e0:::obj23:head# 2026-02-22T20:49:08.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:08.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 24 % 2 2026-02-22T20:49:08.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:08.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj24 remove 2026-02-22T20:49:08.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:08.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:08.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:08.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:08.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:08.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj24 remove 2026-02-22T20:49:08.531 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:7eacede5:::obj24:head# 2026-02-22T20:49:08.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:08.651 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 25 % 2 2026-02-22T20:49:08.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=1 2026-02-22T20:49:08.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 1 obj25 remove 2026-02-22T20:49:08.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:08.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:08.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T20:49:08.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:08.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:08.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/1 obj25 remove 2026-02-22T20:49:08.815 INFO:tasks.workunit.client.0.trial150.stdout:remove 0#1:7109478f:::obj25:head# 2026-02-22T20:49:08.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:819: TEST_repair_stats_ec: for i in $(seq 1 $REPAIRS) 2026-02-22T20:49:08.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: expr 26 % 2 2026-02-22T20:49:08.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:822: TEST_repair_stats_ec: OSD=0 2026-02-22T20:49:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:823: TEST_repair_stats_ec: _objectstore_tool_nodown td/osd-scrub-repair 0 obj26 remove 2026-02-22T20:49:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:49:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:49:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:49:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:49:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:08.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 obj26 remove 2026-02-22T20:49:09.075 INFO:tasks.workunit.client.0.trial150.stdout:remove 1#1:f3c2593a:::obj26:head# 2026-02-22T20:49:09.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:825: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 1 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:49:09.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:49:09.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:49:09.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:49:09.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:49:09.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:49:09.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:49:09.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:49:09.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:49:09.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:49:09.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:49:09.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:49:09.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:49:09.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:49:09.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:49:09.160 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:49:09.160 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:49:09.160 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:49:09.160 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:49:09.160 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:49:09.160 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:49:09.160 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:49:09.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:49:09.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:49:09.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:49:09.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:49:09.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:49:09.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:49:09.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:49:09.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:49:09.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:49:09.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:49:09.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:49:09.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:49:09.162 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:49:09.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:49:09.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/1/whoami 2026-02-22T20:49:09.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:49:09.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:49:09.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:49:09.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:49:09.179 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:09.171+0000 7f96136b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:49:09.181 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:09.171+0000 7f96136b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:49:09.182 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:09.171+0000 7f96136b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:49:09.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:49:09.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:49:09.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:49:09.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:49:09.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:49:09.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:49:09.325 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:49:09.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:49:09.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:49:09.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:49:09.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:49:09.592 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:09.583+0000 7f96136b7a40 -1 Falling back to public interface 2026-02-22T20:49:10.218 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:10.207+0000 7f96136b7a40 -1 osd.1 22 log_to_monitors true 2026-02-22T20:49:10.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:49:10.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:49:10.485 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:49:10.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:49:10.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:49:10.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:49:10.644 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 23 last_clean_interval [10,22) [v2:127.0.0.1:6802/3544722416,v1:127.0.0.1:6803/3544722416] [v2:127.0.0.1:6804/3544722416,v1:127.0.0.1:6805/3544722416] exists,up 7af23ca5-ba78-49ad-8390-906c5e1d1d71 2026-02-22T20:49:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:49:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:49:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:49:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:826: TEST_repair_stats_ec: activate_osd td/osd-scrub-repair 0 --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:49:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:49:10.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:49:10.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:49:10.647 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:49:10.647 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:49:10.647 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:49:10.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:49:10.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:49:10.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:49:10.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:49:10.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:49:10.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:49:10.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:49:10.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:49:10.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:49:10.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:49:10.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0' 2026-02-22T20:49:10.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:49:10.648 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:49:10.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:49:10.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd-scrub-interval-randomize-ratio=0 2026-02-22T20:49:10.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:49:10.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:49:10.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:49:10.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:49:10.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:49:10.673 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:10.663+0000 7fab8c5e5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:49:10.674 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:10.663+0000 7fab8c5e5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:49:10.675 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:10.667+0000 7fab8c5e5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:49:10.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:49:10.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:49:10.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:49:10.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:49:10.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:49:10.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:49:10.812 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:49:10.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:49:10.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:49:10.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:49:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:49:11.008 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:10.999+0000 7fab8c5e5a40 -1 Falling back to public interface 2026-02-22T20:49:11.666 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:11.655+0000 7fab8c5e5a40 -1 osd.0 21 log_to_monitors true 2026-02-22T20:49:11.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:49:11.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:49:11.967 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:49:11.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:49:11.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:49:11.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:49:12.121 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 30 up_thru 0 down_at 21 last_clean_interval [5,21) [v2:127.0.0.1:6810/385018448,v1:127.0.0.1:6811/385018448] [v2:127.0.0.1:6812/385018448,v1:127.0.0.1:6813/385018448] exists,up e3d32c08-79e4-422e-aa50-e0947ef8188c 2026-02-22T20:49:12.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:49:12.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:49:12.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:49:12.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:827: TEST_repair_stats_ec: wait_for_clean 2026-02-22T20:49:12.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:49:12.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:49:12.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:49:12.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:49:12.123 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:49:12.123 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:49:12.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:49:12.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:49:12.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:49:12.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:49:12.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:49:12.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:49:12.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:49:12.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:49:12.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:49:12.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:49:12.401 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:49:12.401 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:49:12.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:49:12.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:49:12.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:49:12.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-22T20:49:12.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-22T20:49:12.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882' 2026-02-22T20:49:12.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:49:12.475 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:49:12.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149698 2026-02-22T20:49:12.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149698 2026-02-22T20:49:12.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-111669149698' 2026-02-22T20:49:12.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:49:12.546 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:49:12.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-22T20:49:12.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-22T20:49:12.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-111669149698 2-60129542150' 2026-02-22T20:49:12.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:49:12.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018882 2026-02-22T20:49:12.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:49:12.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:49:12.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018882 2026-02-22T20:49:12.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:49:12.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-22T20:49:12.626 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 128849018882 2026-02-22T20:49:12.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2026-02-22T20:49:12.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:49:12.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 128849018882 2026-02-22T20:49:12.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:49:13.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:49:13.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:49:13.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018882 -lt 128849018882 2026-02-22T20:49:13.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:49:13.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-111669149698 2026-02-22T20:49:13.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:49:13.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:49:13.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-111669149698 2026-02-22T20:49:13.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:49:13.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=111669149698 2026-02-22T20:49:13.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 111669149698' 2026-02-22T20:49:13.957 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 111669149698 2026-02-22T20:49:13.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:49:14.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 111669149698 -lt 111669149698 2026-02-22T20:49:14.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:49:14.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-22T20:49:14.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:49:14.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:49:14.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542150 2026-02-22T20:49:14.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:49:14.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542150 2026-02-22T20:49:14.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-22T20:49:14.130 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542150 2026-02-22T20:49:14.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:49:14.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542150 -lt 60129542150 2026-02-22T20:49:14.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:49:14.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:49:14.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:49:14.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:49:14.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:49:14.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:49:14.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:49:14.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:49:14.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:49:14.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:49:14.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:49:14.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:49:14.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:49:14.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:49:14.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:49:14.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:49:14.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:49:14.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:49:14.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:829: TEST_repair_stats_ec: repair 1.0 2026-02-22T20:49:14.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-22T20:49:14.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-22T20:49:14.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:49:14.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:14.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:14.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:49:14.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:48:54.904488+0000 2026-02-22T20:49:14.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-22T20:49:15.137 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0s0 on osd.1 to repair 2026-02-22T20:49:15.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-22T20:48:54.904488+0000 2026-02-22T20:49:15.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:49:15.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:48:54.904488+0000 2026-02-22T20:49:15.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:49:15.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:49:15.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:49:15.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:49:15.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:49:15.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:15.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:15.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:49:15.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:48:54.904488+0000 '>' 2026-02-22T20:48:54.904488+0000 2026-02-22T20:49:15.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:49:16.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:49:16.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:49:16.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:49:16.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:49:16.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:16.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:16.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:49:16.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:49:15.204374+0000 '>' 2026-02-22T20:48:54.904488+0000 2026-02-22T20:49:16.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:49:16.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:830: TEST_repair_stats_ec: wait_for_clean 2026-02-22T20:49:16.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:49:16.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:49:16.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:49:16.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:49:16.468 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:49:16.468 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:49:16.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:49:16.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:49:16.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:49:16.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:49:16.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:49:16.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:49:16.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:49:16.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:49:16.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:49:16.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:49:16.737 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:49:16.737 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:49:16.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:49:16.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:49:16.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:49:16.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018884 2026-02-22T20:49:16.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018884 2026-02-22T20:49:16.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884' 2026-02-22T20:49:16.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:49:16.810 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:49:16.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149700 2026-02-22T20:49:16.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149700 2026-02-22T20:49:16.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884 1-111669149700' 2026-02-22T20:49:16.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:49:16.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:49:16.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542152 2026-02-22T20:49:16.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542152 2026-02-22T20:49:16.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884 1-111669149700 2-60129542152' 2026-02-22T20:49:16.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:49:16.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018884 2026-02-22T20:49:16.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:49:16.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:49:16.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018884 2026-02-22T20:49:16.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:49:16.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018884 2026-02-22T20:49:16.959 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 128849018884 2026-02-22T20:49:16.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018884' 2026-02-22T20:49:16.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:49:17.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018882 -lt 128849018884 2026-02-22T20:49:17.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:49:18.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:49:18.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:49:18.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018884 2026-02-22T20:49:18.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:49:18.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-111669149700 2026-02-22T20:49:18.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:49:18.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:49:18.280 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-111669149700 2026-02-22T20:49:18.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:49:18.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=111669149700 2026-02-22T20:49:18.283 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 111669149700 2026-02-22T20:49:18.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 111669149700' 2026-02-22T20:49:18.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:49:18.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 111669149700 -lt 111669149700 2026-02-22T20:49:18.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:49:18.449 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542152 2026-02-22T20:49:18.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:49:18.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:49:18.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542152 2026-02-22T20:49:18.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:49:18.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542152 2026-02-22T20:49:18.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2026-02-22T20:49:18.454 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542152 2026-02-22T20:49:18.455 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:49:18.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542152 -lt 60129542152 2026-02-22T20:49:18.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:49:18.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:49:18.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:49:18.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:49:18.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:49:18.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:49:18.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:49:18.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:49:18.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:49:18.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:49:18.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:49:18.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:49:18.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:49:18.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:49:18.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:49:19.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:49:19.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:49:19.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:49:19.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:831: TEST_repair_stats_ec: ceph pg dump pgs 2026-02-22T20:49:19.299 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:49:19.299 INFO:tasks.workunit.client.0.trial150.stdout:1.0 30 0 0 0 0 210 0 0 30 0 30 active+clean 2026-02-22T20:49:15.232897+0000 20'30 31:139 [1,0,2] 1 [1,0,2] 1 20'30 2026-02-22T20:49:15.204374+0000 20'30 2026-02-22T20:49:15.204374+0000 0 1 periodic scrub scheduled @ 2026-02-23T20:49:15.204374+0000 17 0 2026-02-22T20:49:19.299 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:49:19.299 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:49:19.300 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:49:19.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:832: TEST_repair_stats_ec: flush_pg_stats 2026-02-22T20:49:19.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:49:19.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:49:19.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:49:19.469 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:49:19.469 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:49:19.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:49:19.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:49:19.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:49:19.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018885 2026-02-22T20:49:19.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018885 2026-02-22T20:49:19.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018885' 2026-02-22T20:49:19.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:49:19.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:49:19.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=111669149701 2026-02-22T20:49:19.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 111669149701 2026-02-22T20:49:19.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018885 1-111669149701' 2026-02-22T20:49:19.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:49:19.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:49:19.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542154 2026-02-22T20:49:19.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542154 2026-02-22T20:49:19.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018885 1-111669149701 2-60129542154' 2026-02-22T20:49:19.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:49:19.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018885 2026-02-22T20:49:19.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:49:19.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:49:19.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018885 2026-02-22T20:49:19.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:49:19.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018885 2026-02-22T20:49:19.698 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 128849018885 2026-02-22T20:49:19.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018885' 2026-02-22T20:49:19.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:49:19.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018885 2026-02-22T20:49:19.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:49:20.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:49:20.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:49:21.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018885 -lt 128849018885 2026-02-22T20:49:21.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:49:21.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-111669149701 2026-02-22T20:49:21.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:49:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:49:21.023 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-111669149701 2026-02-22T20:49:21.024 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:49:21.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=111669149701 2026-02-22T20:49:21.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 111669149701' 2026-02-22T20:49:21.026 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 111669149701 2026-02-22T20:49:21.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:49:21.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 111669149701 -lt 111669149701 2026-02-22T20:49:21.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:49:21.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542154 2026-02-22T20:49:21.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:49:21.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:49:21.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542154 2026-02-22T20:49:21.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:49:21.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542154 2026-02-22T20:49:21.203 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542154 2026-02-22T20:49:21.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542154' 2026-02-22T20:49:21.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:49:21.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542154 -lt 60129542154 2026-02-22T20:49:21.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: TEST_repair_stats_ec: ceph pg 1.0 query 2026-02-22T20:49:21.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:835: TEST_repair_stats_ec: jq .info.stats.stat_sum 2026-02-22T20:49:21.438 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:21.438 INFO:tasks.workunit.client.0.trial150.stdout: "num_bytes": 210, 2026-02-22T20:49:21.438 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects": 30, 2026-02-22T20:49:21.438 INFO:tasks.workunit.client.0.trial150.stdout: "num_object_clones": 0, 2026-02-22T20:49:21.438 INFO:tasks.workunit.client.0.trial150.stdout: "num_object_copies": 90, 2026-02-22T20:49:21.438 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_missing_on_primary": 0, 2026-02-22T20:49:21.438 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_missing": 0, 2026-02-22T20:49:21.438 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_degraded": 0, 2026-02-22T20:49:21.438 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_misplaced": 0, 2026-02-22T20:49:21.438 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_unfound": 0, 2026-02-22T20:49:21.438 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_dirty": 30, 2026-02-22T20:49:21.439 INFO:tasks.workunit.client.0.trial150.stdout: "num_whiteouts": 0, 2026-02-22T20:49:21.439 INFO:tasks.workunit.client.0.trial150.stdout: "num_read": 0, 2026-02-22T20:49:21.439 INFO:tasks.workunit.client.0.trial150.stdout: "num_read_kb": 0, 2026-02-22T20:49:21.439 INFO:tasks.workunit.client.0.trial150.stdout: "num_write": 30, 2026-02-22T20:49:21.439 INFO:tasks.workunit.client.0.trial150.stdout: "num_write_kb": 30, 2026-02-22T20:49:21.439 INFO:tasks.workunit.client.0.trial150.stdout: "num_scrub_errors": 0, 2026-02-22T20:49:21.439 INFO:tasks.workunit.client.0.trial150.stdout: "num_shallow_scrub_errors": 0, 2026-02-22T20:49:21.439 INFO:tasks.workunit.client.0.trial150.stdout: "num_deep_scrub_errors": 0, 2026-02-22T20:49:21.439 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_recovered": 26, 2026-02-22T20:49:21.439 INFO:tasks.workunit.client.0.trial150.stdout: "num_bytes_recovered": 182, 2026-02-22T20:49:21.439 INFO:tasks.workunit.client.0.trial150.stdout: "num_keys_recovered": 0, 2026-02-22T20:49:21.439 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_omap": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_hit_set_archive": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_bytes_hit_set_archive": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_flush": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_flush_kb": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_evict": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_evict_kb": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_promote": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_flush_mode_high": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_flush_mode_low": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_evict_mode_some": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_evict_mode_full": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_pinned": 0, 2026-02-22T20:49:21.440 INFO:tasks.workunit.client.0.trial150.stdout: "num_legacy_snapsets": 0, 2026-02-22T20:49:21.441 INFO:tasks.workunit.client.0.trial150.stdout: "num_large_omap_objects": 0, 2026-02-22T20:49:21.441 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_manifest": 0, 2026-02-22T20:49:21.441 INFO:tasks.workunit.client.0.trial150.stdout: "num_omap_bytes": 0, 2026-02-22T20:49:21.441 INFO:tasks.workunit.client.0.trial150.stdout: "num_omap_keys": 0, 2026-02-22T20:49:21.441 INFO:tasks.workunit.client.0.trial150.stdout: "num_objects_repaired": 26 2026-02-22T20:49:21.441 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:21.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: TEST_repair_stats_ec: ceph pg 1.0 query 2026-02-22T20:49:21.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: TEST_repair_stats_ec: jq .info.stats.stat_sum.num_objects_repaired 2026-02-22T20:49:21.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:836: TEST_repair_stats_ec: COUNT=26 2026-02-22T20:49:21.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:837: TEST_repair_stats_ec: test 26 = 26 2026-02-22T20:49:21.515 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: expr 3 - 1 2026-02-22T20:49:21.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: seq 0 2 2026-02-22T20:49:21.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:49:21.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: TEST_repair_stats_ec: '[' 0 = 0 -o 0 = 1 ']' 2026-02-22T20:49:21.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:841: TEST_repair_stats_ec: expr 26 / 2 2026-02-22T20:49:21.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:841: TEST_repair_stats_ec: repair=13 2026-02-22T20:49:21.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-22T20:49:21.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 )' 2026-02-22T20:49:21.662 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:49:21.670 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:21.670 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:49:21.670 INFO:tasks.workunit.client.0.trial150.stdout: "up_from": 30, 2026-02-22T20:49:21.670 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 128849018885, 2026-02-22T20:49:21.670 INFO:tasks.workunit.client.0.trial150.stdout: "num_pgs": 1, 2026-02-22T20:49:21.670 INFO:tasks.workunit.client.0.trial150.stdout: "num_osds": 1, 2026-02-22T20:49:21.670 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_osds": 1, 2026-02-22T20:49:21.670 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_omap_osds": 1, 2026-02-22T20:49:21.670 INFO:tasks.workunit.client.0.trial150.stdout: "kb": 104857600, 2026-02-22T20:49:21.670 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used": 28488, 2026-02-22T20:49:21.670 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_data": 376, 2026-02-22T20:49:21.670 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_omap": 5, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_meta": 28090, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "kb_avail": 104829112, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "statfs": { 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "total": 107374182400, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "available": 107345010688, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "internally_reserved": 0, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "allocated": 385024, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "data_stored": 206161, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed": 0, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_allocated": 0, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_original": 0, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "omap_allocated": 5703, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "internal_metadata": 28764601 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "hb_peers": [ 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: 1, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: 2 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "snap_trim_queue_len": 0, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "num_snap_trimming": 0, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "num_shards_repaired": 13, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "op_queue_age_hist": { 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "histogram": [], 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "upper_bound": 1 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "perf_stat": { 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ms": 0, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ms": 0, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ns": 0, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ns": 0 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout: "alerts": [] 2026-02-22T20:49:21.671 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:21.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-22T20:49:21.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 0 ).num_shards_repaired' 2026-02-22T20:49:21.820 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:49:21.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: COUNT=13 2026-02-22T20:49:21.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:848: TEST_repair_stats_ec: test 13 = 13 2026-02-22T20:49:21.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:49:21.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: TEST_repair_stats_ec: '[' 1 = 0 -o 1 = 1 ']' 2026-02-22T20:49:21.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:841: TEST_repair_stats_ec: expr 26 / 2 2026-02-22T20:49:21.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:841: TEST_repair_stats_ec: repair=13 2026-02-22T20:49:21.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-22T20:49:21.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 )' 2026-02-22T20:49:21.980 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:49:21.988 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:21.989 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:49:21.989 INFO:tasks.workunit.client.0.trial150.stdout: "up_from": 26, 2026-02-22T20:49:21.989 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 111669149702, 2026-02-22T20:49:21.989 INFO:tasks.workunit.client.0.trial150.stdout: "num_pgs": 1, 2026-02-22T20:49:21.989 INFO:tasks.workunit.client.0.trial150.stdout: "num_osds": 1, 2026-02-22T20:49:21.989 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_osds": 1, 2026-02-22T20:49:21.989 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_omap_osds": 1, 2026-02-22T20:49:21.989 INFO:tasks.workunit.client.0.trial150.stdout: "kb": 104857600, 2026-02-22T20:49:21.989 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used": 28488, 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_data": 376, 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_omap": 7, 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_meta": 28088, 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "kb_avail": 104829112, 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "statfs": { 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "total": 107374182400, 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "available": 107345010688, 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "internally_reserved": 0, 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "allocated": 385024, 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "data_stored": 206161, 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed": 0, 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_allocated": 0, 2026-02-22T20:49:21.990 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_original": 0, 2026-02-22T20:49:21.991 INFO:tasks.workunit.client.0.trial150.stdout: "omap_allocated": 7655, 2026-02-22T20:49:21.991 INFO:tasks.workunit.client.0.trial150.stdout: "internal_metadata": 28762649 2026-02-22T20:49:21.991 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:49:21.991 INFO:tasks.workunit.client.0.trial150.stdout: "hb_peers": [ 2026-02-22T20:49:21.991 INFO:tasks.workunit.client.0.trial150.stdout: 0, 2026-02-22T20:49:21.991 INFO:tasks.workunit.client.0.trial150.stdout: 2 2026-02-22T20:49:21.991 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:49:21.991 INFO:tasks.workunit.client.0.trial150.stdout: "snap_trim_queue_len": 0, 2026-02-22T20:49:21.991 INFO:tasks.workunit.client.0.trial150.stdout: "num_snap_trimming": 0, 2026-02-22T20:49:21.991 INFO:tasks.workunit.client.0.trial150.stdout: "num_shards_repaired": 13, 2026-02-22T20:49:21.991 INFO:tasks.workunit.client.0.trial150.stdout: "op_queue_age_hist": { 2026-02-22T20:49:21.991 INFO:tasks.workunit.client.0.trial150.stdout: "histogram": [], 2026-02-22T20:49:21.992 INFO:tasks.workunit.client.0.trial150.stdout: "upper_bound": 1 2026-02-22T20:49:21.992 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:49:21.992 INFO:tasks.workunit.client.0.trial150.stdout: "perf_stat": { 2026-02-22T20:49:21.992 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ms": 0, 2026-02-22T20:49:21.992 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ms": 0, 2026-02-22T20:49:21.992 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ns": 0, 2026-02-22T20:49:21.992 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ns": 0 2026-02-22T20:49:21.992 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:49:21.992 INFO:tasks.workunit.client.0.trial150.stdout: "alerts": [] 2026-02-22T20:49:21.992 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:21.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-22T20:49:21.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 1 ).num_shards_repaired' 2026-02-22T20:49:22.138 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:49:22.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: COUNT=13 2026-02-22T20:49:22.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:848: TEST_repair_stats_ec: test 13 = 13 2026-02-22T20:49:22.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:839: TEST_repair_stats_ec: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:49:22.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:840: TEST_repair_stats_ec: '[' 2 = 0 -o 2 = 1 ']' 2026-02-22T20:49:22.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:843: TEST_repair_stats_ec: repair=0 2026-02-22T20:49:22.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-22T20:49:22.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:846: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 )' 2026-02-22T20:49:22.296 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 2, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "up_from": 14, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 60129542154, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "num_pgs": 1, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "num_osds": 1, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_osds": 1, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_omap_osds": 1, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "kb": 104857600, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used": 27336, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_data": 376, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_omap": 13, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_meta": 26930, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "kb_avail": 104830264, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "statfs": { 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "total": 107374182400, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "available": 107346190336, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "internally_reserved": 0, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "allocated": 385024, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "data_stored": 206161, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed": 0, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_allocated": 0, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_original": 0, 2026-02-22T20:49:22.305 INFO:tasks.workunit.client.0.trial150.stdout: "omap_allocated": 13343, 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "internal_metadata": 27577313 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "hb_peers": [ 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: 0, 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "snap_trim_queue_len": 0, 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "num_snap_trimming": 0, 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "num_shards_repaired": 0, 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "op_queue_age_hist": { 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "histogram": [], 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "upper_bound": 1 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "perf_stat": { 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ms": 0, 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ms": 0, 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ns": 0, 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ns": 0 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout: "alerts": [] 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-22T20:49:22.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: jq '.pg_map.osd_stats[] | select(.osd == 2 ).num_shards_repaired' 2026-02-22T20:49:22.453 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:49:22.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:847: TEST_repair_stats_ec: COUNT=0 2026-02-22T20:49:22.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:848: TEST_repair_stats_ec: test 0 = 0 2026-02-22T20:49:22.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:851: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-22T20:49:22.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:851: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum 2026-02-22T20:49:22.613 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:49:22.624 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:22.624 INFO:tasks.workunit.client.0.trial150.stdout: "up_from": 0, 2026-02-22T20:49:22.624 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 0, 2026-02-22T20:49:22.624 INFO:tasks.workunit.client.0.trial150.stdout: "num_pgs": 3, 2026-02-22T20:49:22.625 INFO:tasks.workunit.client.0.trial150.stdout: "num_osds": 3, 2026-02-22T20:49:22.625 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_osds": 3, 2026-02-22T20:49:22.625 INFO:tasks.workunit.client.0.trial150.stdout: "num_per_pool_omap_osds": 3, 2026-02-22T20:49:22.625 INFO:tasks.workunit.client.0.trial150.stdout: "kb": 314572800, 2026-02-22T20:49:22.625 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used": 84312, 2026-02-22T20:49:22.625 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_data": 1128, 2026-02-22T20:49:22.625 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_omap": 26, 2026-02-22T20:49:22.625 INFO:tasks.workunit.client.0.trial150.stdout: "kb_used_meta": 83109, 2026-02-22T20:49:22.625 INFO:tasks.workunit.client.0.trial150.stdout: "kb_avail": 314488488, 2026-02-22T20:49:22.625 INFO:tasks.workunit.client.0.trial150.stdout: "statfs": { 2026-02-22T20:49:22.625 INFO:tasks.workunit.client.0.trial150.stdout: "total": 322122547200, 2026-02-22T20:49:22.625 INFO:tasks.workunit.client.0.trial150.stdout: "available": 322036211712, 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: "internally_reserved": 0, 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: "allocated": 1155072, 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: "data_stored": 618483, 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed": 0, 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_allocated": 0, 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: "data_compressed_original": 0, 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: "omap_allocated": 26701, 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: "internal_metadata": 85104563 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: "hb_peers": [], 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: "snap_trim_queue_len": 0, 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: "num_snap_trimming": 0, 2026-02-22T20:49:22.626 INFO:tasks.workunit.client.0.trial150.stdout: "num_shards_repaired": 26, 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout: "op_queue_age_hist": { 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout: "histogram": [], 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout: "upper_bound": 1 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout: "perf_stat": { 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ms": 0, 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ms": 0, 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout: "commit_latency_ns": 0, 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout: "apply_latency_ns": 0 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout: "alerts": [], 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout: "network_ping_times": [] 2026-02-22T20:49:22.627 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:22.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: TEST_repair_stats_ec: ceph pg dump --format=json-pretty 2026-02-22T20:49:22.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: TEST_repair_stats_ec: jq .pg_map.osd_stats_sum.num_shards_repaired 2026-02-22T20:49:22.775 INFO:tasks.workunit.client.0.trial150.stderr:dumped all 2026-02-22T20:49:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:852: TEST_repair_stats_ec: COUNT=26 2026-02-22T20:49:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:853: TEST_repair_stats_ec: test 26 = 26 2026-02-22T20:49:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:49:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:49:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:49:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:49:22.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:49:22.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:49:22.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:49:22.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:49:22.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:49:22.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:49:22.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:49:22.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:49:22.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:49:22.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:49:22.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:49:22.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:49:22.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:49:22.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:49:22.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:49:22.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:49:22.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:49:22.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:49:22.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:49:22.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:49:22.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:49:22.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:49:22.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:49:22.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:49:22.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:49:22.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:49:22.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:49:22.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:49:22.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:49:22.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:49:22.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:49:22.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:49:22.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:49:22.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:49:22.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:49:22.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:49:22.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:49:22.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:49:22.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:49:22.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:49:22.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:49:22.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:49:22.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:49:22.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:49:22.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:49:22.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:49:22.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:49:22.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:49:22.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:49:22.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:49:22.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:49:22.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:49:22.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:49:22.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:49:22.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:49:22.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:49:22.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:49:22.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:49:22.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:49:22.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:49:22.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:49:22.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:49:22.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:49:22.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:49:22.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:49:22.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:49:22.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_request_scrub_priority td/osd-scrub-repair 2026-02-22T20:49:22.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6369: TEST_request_scrub_priority: local dir=td/osd-scrub-repair 2026-02-22T20:49:22.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6370: TEST_request_scrub_priority: local poolname=psr_pool 2026-02-22T20:49:22.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6371: TEST_request_scrub_priority: local objname=POBJ 2026-02-22T20:49:22.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6372: TEST_request_scrub_priority: local OBJECTS=64 2026-02-22T20:49:22.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6373: TEST_request_scrub_priority: local PGS=8 2026-02-22T20:49:22.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6375: TEST_request_scrub_priority: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T20:49:22.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:49:22.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:49:22.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:49:22.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:49:22.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:49:22.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T20:49:22.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:49:22.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:49:22.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:49:22.966 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:49:22.967 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:49:22.967 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:49:22.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:49:22.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T20:49:22.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:49:22.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:49:22.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:49:22.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:49:22.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:49:22.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:49:22.998 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:49:22.998 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:49:22.998 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:49:22.999 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:49:22.999 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:49:22.999 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:49:22.999 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:49:22.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:49:22.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:49:23.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:49:23.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:49:23.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:49:23.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:49:23.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:49:23.054 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:49:23.054 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:49:23.054 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:49:23.055 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:49:23.055 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:49:23.055 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:49:23.055 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:49:23.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:49:23.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:49:23.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6376: TEST_request_scrub_priority: run_mgr td/osd-scrub-repair x 2026-02-22T20:49:23.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:49:23.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:49:23.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:49:23.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:49:23.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:49:23.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:49:23.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:49:23.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:49:23.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:49:23.204 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:49:23.204 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:49:23.204 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:49:23.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:49:23.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:49:23.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:49:23.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6377: TEST_request_scrub_priority: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2026-02-22T20:49:23.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6378: TEST_request_scrub_priority: ceph_osd_args+=--osd_scrub_backoff_ratio=0 2026-02-22T20:49:23.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6379: TEST_request_scrub_priority: run_osd td/osd-scrub-repair 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2026-02-22T20:49:23.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:49:23.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:49:23.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:49:23.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:49:23.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:49:23.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:49:23.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:49:23.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:49:23.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:49:23.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:49:23.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:49:23.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:49:23.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:49:23.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:49:23.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:49:23.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:49:23.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:49:23.231 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:49:23.231 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:49:23.231 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:49:23.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:49:23.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:49:23.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:49:23.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:49:23.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:49:23.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:49:23.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:49:23.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:49:23.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:49:23.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:49:23.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:49:23.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:49:23.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0' 2026-02-22T20:49:23.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:49:23.233 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:49:23.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c86b56a7-f053-4fde-84fc-9621e7d81456 2026-02-22T20:49:23.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c86b56a7-f053-4fde-84fc-9621e7d81456' 2026-02-22T20:49:23.234 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 c86b56a7-f053-4fde-84fc-9621e7d81456 2026-02-22T20:49:23.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:49:23.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDTa5tp0ssJDhAAEJtMEpsOAikA04Rb/hVn5A== 2026-02-22T20:49:23.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDTa5tp0ssJDhAAEJtMEpsOAikA04Rb/hVn5A=="}' 2026-02-22T20:49:23.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c86b56a7-f053-4fde-84fc-9621e7d81456 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:49:23.334 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:49:23.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:49:23.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --mkfs --key AQDTa5tp0ssJDhAAEJtMEpsOAikA04Rb/hVn5A== --osd-uuid c86b56a7-f053-4fde-84fc-9621e7d81456 2026-02-22T20:49:23.359 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:23.351+0000 7fa2c8637a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:49:23.361 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:23.351+0000 7fa2c8637a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:49:23.362 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:23.351+0000 7fa2c8637a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:49:23.362 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:23.351+0000 7fa2c8637a40 -1 bdev(0x55ca90f26c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:49:23.362 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:23.351+0000 7fa2c8637a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:49:24.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:49:24.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:49:24.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:49:24.131 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:49:24.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:49:24.338 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:49:24.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:49:24.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 2026-02-22T20:49:24.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:49:24.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:49:24.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:49:24.355 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:24.347+0000 7fce72e9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:49:24.357 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:24.347+0000 7fce72e9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:49:24.358 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:24.347+0000 7fce72e9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:49:24.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:49:24.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:49:24.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:49:24.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:49:24.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:49:24.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:49:24.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:49:24.501 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:49:24.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:49:24.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:49:24.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:49:24.740 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:24.731+0000 7fce72e9da40 -1 Falling back to public interface 2026-02-22T20:49:25.375 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:49:25.367+0000 7fce72e9da40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:49:25.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:49:25.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:49:25.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:49:25.665 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:49:25.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:49:25.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:49:25.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:49:26.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:49:26.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:49:26.826 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:49:26.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:49:26.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:49:26.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:49:26.982 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3640998163,v1:127.0.0.1:6803/3640998163] [v2:127.0.0.1:6804/3640998163,v1:127.0.0.1:6805/3640998163] exists,up c86b56a7-f053-4fde-84fc-9621e7d81456 2026-02-22T20:49:26.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:49:26.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:49:26.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:49:26.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6381: TEST_request_scrub_priority: create_pool psr_pool 8 8 2026-02-22T20:49:26.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool 8 8 2026-02-22T20:49:27.181 INFO:tasks.workunit.client.0.trial150.stderr:pool 'psr_pool' created 2026-02-22T20:49:27.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:49:28.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6382: TEST_request_scrub_priority: wait_for_clean 2026-02-22T20:49:28.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:49:28.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:49:28.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:49:28.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:49:28.194 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:49:28.194 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:49:28.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:49:28.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:49:28.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:49:28.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:49:28.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:49:28.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:49:28.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:49:28.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:49:28.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:49:28.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:49:28.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:49:28.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:49:28.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:49:28.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836482 2026-02-22T20:49:28.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836482 2026-02-22T20:49:28.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836482' 2026-02-22T20:49:28.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:49:28.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836482 2026-02-22T20:49:28.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:49:28.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:49:28.552 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836482 2026-02-22T20:49:28.552 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:49:28.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836482 2026-02-22T20:49:28.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2026-02-22T20:49:28.554 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836482 2026-02-22T20:49:28.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:49:28.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836482 2026-02-22T20:49:28.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:49:29.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:49:29.720 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:49:29.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836482 2026-02-22T20:49:29.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:49:29.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:49:29.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:49:30.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 8 == 0 2026-02-22T20:49:30.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:49:30.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:49:30.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:49:30.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:49:30.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:49:30.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:49:30.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:49:30.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=8 2026-02-22T20:49:30.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:49:30.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:49:30.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:49:30.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 = 8 2026-02-22T20:49:30.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:49:30.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:49:30.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6384: TEST_request_scrub_priority: local osd=0 2026-02-22T20:49:30.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6385: TEST_request_scrub_priority: add_something td/osd-scrub-repair psr_pool POBJ noscrub 2026-02-22T20:49:30.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:49:30.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=psr_pool 2026-02-22T20:49:30.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=POBJ 2026-02-22T20:49:30.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:49:30.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:49:30.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:49:30.626 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:49:30.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:49:30.831 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:49:30.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:49:30.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:49:30.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool psr_pool put POBJ td/osd-scrub-repair/ORIGINAL 2026-02-22T20:49:30.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6386: TEST_request_scrub_priority: get_primary psr_pool POBJ 2026-02-22T20:49:30.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=psr_pool 2026-02-22T20:49:30.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=POBJ 2026-02-22T20:49:30.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map psr_pool POBJ 2026-02-22T20:49:30.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:49:31.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6386: TEST_request_scrub_priority: local primary=0 2026-02-22T20:49:31.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6387: TEST_request_scrub_priority: get_pg psr_pool POBJ 2026-02-22T20:49:31.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=psr_pool 2026-02-22T20:49:31.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=POBJ 2026-02-22T20:49:31.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map psr_pool POBJ 2026-02-22T20:49:31.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:49:31.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6387: TEST_request_scrub_priority: local pg=1.1 2026-02-22T20:49:31.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6388: TEST_request_scrub_priority: ceph osd dump 2026-02-22T20:49:31.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6388: TEST_request_scrub_priority: grep '^pool.*['\'']psr_pool['\'']' 2026-02-22T20:49:31.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6388: TEST_request_scrub_priority: awk '{ print $2 }' 2026-02-22T20:49:31.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6388: TEST_request_scrub_priority: poolid=1 2026-02-22T20:49:31.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6390: TEST_request_scrub_priority: local otherpgs 2026-02-22T20:49:31.346 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: expr 8 - 1 2026-02-22T20:49:31.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: seq 0 7 2026-02-22T20:49:31.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-22T20:49:31.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.0 2026-02-22T20:49:31.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.0 = 1.1 ']' 2026-02-22T20:49:31.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 ' 2026-02-22T20:49:31.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-22T20:49:31.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-22T20:49:31.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:31.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:31.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-22T20:49:31.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:31.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.0 schedule-scrub 1.0 2026-02-22T20:49:31.576 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:31.576 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:49:31.576 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:49:31.576 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-20T20:49:31.569202+0000" 2026-02-22T20:49:31.576 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:31.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-22T20:49:31.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.1 2026-02-22T20:49:31.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.1 = 1.1 ']' 2026-02-22T20:49:31.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6395: TEST_request_scrub_priority: continue 2026-02-22T20:49:31.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-22T20:49:31.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.2 2026-02-22T20:49:31.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.2 = 1.1 ']' 2026-02-22T20:49:31.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 ' 2026-02-22T20:49:31.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-22T20:49:31.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-22T20:49:31.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:31.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:31.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-22T20:49:31.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:31.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.2 schedule-scrub 1.2 2026-02-22T20:49:31.812 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:31.812 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:49:31.812 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:49:31.812 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-20T20:49:31.805017+0000" 2026-02-22T20:49:31.812 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:31.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-22T20:49:31.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.3 2026-02-22T20:49:31.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.3 = 1.1 ']' 2026-02-22T20:49:31.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 ' 2026-02-22T20:49:31.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-22T20:49:31.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-22T20:49:31.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:31.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:31.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-22T20:49:31.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:31.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.3 schedule-scrub 1.3 2026-02-22T20:49:32.049 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:32.049 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:49:32.049 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:49:32.049 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-20T20:49:32.041908+0000" 2026-02-22T20:49:32.049 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:32.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-22T20:49:32.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.4 2026-02-22T20:49:32.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.4 = 1.1 ']' 2026-02-22T20:49:32.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 ' 2026-02-22T20:49:32.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-22T20:49:32.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-22T20:49:32.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:32.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:32.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-22T20:49:32.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:32.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.4 schedule-scrub 1.4 2026-02-22T20:49:32.288 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:32.289 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:49:32.289 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:49:32.289 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-20T20:49:32.281755+0000" 2026-02-22T20:49:32.289 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:32.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-22T20:49:32.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.5 2026-02-22T20:49:32.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.5 = 1.1 ']' 2026-02-22T20:49:32.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 ' 2026-02-22T20:49:32.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-22T20:49:32.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-22T20:49:32.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:32.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:32.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-22T20:49:32.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:32.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.5 schedule-scrub 1.5 2026-02-22T20:49:32.527 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:32.527 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:49:32.527 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:49:32.527 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-20T20:49:32.520342+0000" 2026-02-22T20:49:32.527 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:32.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-22T20:49:32.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.6 2026-02-22T20:49:32.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.6 = 1.1 ']' 2026-02-22T20:49:32.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 ' 2026-02-22T20:49:32.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-22T20:49:32.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-22T20:49:32.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:32.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:32.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-22T20:49:32.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:32.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.6 schedule-scrub 1.6 2026-02-22T20:49:32.764 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:32.764 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:49:32.764 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:49:32.764 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-20T20:49:32.757264+0000" 2026-02-22T20:49:32.765 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:32.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6391: TEST_request_scrub_priority: for i in $(seq 0 $(expr $PGS - 1)) 2026-02-22T20:49:32.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6393: TEST_request_scrub_priority: opg=1.7 2026-02-22T20:49:32.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6394: TEST_request_scrub_priority: '[' 1.7 = 1.1 ']' 2026-02-22T20:49:32.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6397: TEST_request_scrub_priority: otherpgs='1.0 1.2 1.3 1.4 1.5 1.6 1.7 ' 2026-02-22T20:49:32.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-22T20:49:32.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-22T20:49:32.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:32.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:32.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-22T20:49:32.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6398: TEST_request_scrub_priority: local other_last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:32.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6400: TEST_request_scrub_priority: ceph tell 1.7 schedule-scrub 1.7 2026-02-22T20:49:32.999 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:32.999 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:49:33.000 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:49:33.000 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-20T20:49:32.992451+0000" 2026-02-22T20:49:33.000 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:33.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6403: TEST_request_scrub_priority: sleep 15 2026-02-22T20:49:48.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6404: TEST_request_scrub_priority: flush_pg_stats 2026-02-22T20:49:48.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:49:48.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:49:48.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:49:48.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:49:48.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:49:48.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:49:48.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-22T20:49:48.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-22T20:49:48.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-22T20:49:48.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:49:48.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-22T20:49:48.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:49:48.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:49:48.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-22T20:49:48.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:49:48.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-22T20:49:48.252 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836487 2026-02-22T20:49:48.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-22T20:49:48.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:49:48.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T20:49:48.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:49:49.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:49:49.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:49:49.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T20:49:49.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:49:50.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:49:50.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:49:50.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-22T20:49:50.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6407: TEST_request_scrub_priority: get_last_scrub_stamp 1.1 2026-02-22T20:49:50.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-22T20:49:50.743 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:50.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:50.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-22T20:49:50.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6407: TEST_request_scrub_priority: local last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:50.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6408: TEST_request_scrub_priority: ceph tell 1.1 scrub 2026-02-22T20:49:50.967 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:49:50.967 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:49:50.968 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:49:50.968 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:49:50.968 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:49:50.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6410: TEST_request_scrub_priority: ceph osd unset noscrub 2026-02-22T20:49:51.189 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is unset 2026-02-22T20:49:51.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6411: TEST_request_scrub_priority: ceph osd unset nodeep-scrub 2026-02-22T20:49:51.396 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is unset 2026-02-22T20:49:51.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6413: TEST_request_scrub_priority: wait_for_scrub 1.1 2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:51.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.1 2026-02-22T20:49:51.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:51.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:49:51.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:49:51.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:49:51.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2026-02-22T20:49:51.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-22T20:49:51.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:51.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:51.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-22T20:49:51.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:49:27.173434+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:51.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:49:52.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:49:52.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:49:52.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2026-02-22T20:49:52.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-22T20:49:52.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:52.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-22T20:49:52.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:52.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:49:51.293624+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:52.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:49:52.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-22T20:49:52.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.0 2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:52.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:49:52.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:52.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:49:52.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:49:52.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:49:52.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:49:52.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:49:52.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:52.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:52.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:49:52.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:52.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:49:53.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:49:53.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:49:53.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:49:53.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:49:53.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:53.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:49:53.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:54.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:54.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:49:55.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:49:55.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:49:55.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:49:55.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:49:55.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:55.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:55.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:49:55.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:55.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:49:56.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:49:56.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:49:56.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:49:56.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:49:56.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:56.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:56.218 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:49:56.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:56.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:49:57.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:49:57.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:49:57.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:49:57.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:49:57.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:57.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:49:57.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:57.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:57.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:49:58.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:49:58.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:49:58.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:49:58.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:49:58.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:58.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:49:58.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:58.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:58.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:49:59.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:49:59.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:49:59.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:49:59.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:49:59.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:49:59.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:49:59.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:49:59.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:49:59.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:00.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:00.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:00.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:00.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:00.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:00.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:00.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:01.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:01.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:02.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:02.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:02.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:02.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:02.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:02.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:02.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:02.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:02.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:03.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:03.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:03.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:03.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:03.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:03.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:03.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:03.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:03.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:04.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:04.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:04.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:04.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:04.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:04.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:04.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:04.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:04.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:05.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:05.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:05.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:05.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:05.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:05.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:05.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:05.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:05.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:06.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:06.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:06.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:06.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:06.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:06.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:06.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:06.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:06.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:07.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:07.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:07.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:07.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:07.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:07.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:07.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:07.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:07.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:08.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:08.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:08.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:08.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:08.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:08.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:08.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:09.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:09.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:10.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:10.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:10.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:10.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:10.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:10.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:10.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:10.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:10.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:11.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:11.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:11.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:11.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:11.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:11.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:11.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:11.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:11.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:12.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:12.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:12.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:12.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:12.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:12.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:12.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:12.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:12.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:13.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:13.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:13.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:13.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:13.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:13.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:13.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:13.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:13.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:14.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:14.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:14.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:14.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:14.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:14.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:14.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:14.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:14.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:15.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:15.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:15.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:15.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:15.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:15.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:15.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:16.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:16.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:17.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:17.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:17.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:17.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:17.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:17.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:17.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:17.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:17.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:18.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:18.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:18.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:18.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:18.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:18.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:18.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:18.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:18.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:19.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:19.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:19.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:19.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:19.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:19.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:19.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:19.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:19.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:20.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:20.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:20.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:20.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:20.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:20.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:20.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:20.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:20.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:21.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:21.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:21.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:21.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:21.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:21.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:21.749 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:21.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:21.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:22.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:22.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:22.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:22.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:22.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:22.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:22.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:23.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:23.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:24.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:24.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:24.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:24.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:24.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:24.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:24.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:24.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:24.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:25.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:25.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:25.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:25.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:25.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:25.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:25.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:25.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:25.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:26.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:26.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:26.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:26.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:26.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:26.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:26.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:26.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:26.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:27.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:27.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:27.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:27.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:27.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:27.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:27.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:27.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:27.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:28.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:28.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:28.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:28.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:28.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:28.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:28.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:28.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:28.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:29.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:29.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:29.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:29.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:29.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:29.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:29.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:30.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:30.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:31.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:31.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:31.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:31.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:31.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:31.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:31.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:31.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:31.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:32.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:32.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:32.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:32.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:32.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:32.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:32.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:32.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:32.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:33.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:33.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:33.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:33.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:33.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:33.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:33.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:33.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:33.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:34.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:34.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:34.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:34.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:34.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:34.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:34.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:34.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:34.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:35.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:35.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:35.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:35.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:35.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:35.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:35.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:35.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:35.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:36.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:36.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:36.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:36.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:36.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:36.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:36.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:37.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:31.569202+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:37.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:38.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:38.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:38.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:50:38.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:50:38.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:38.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:50:38.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:38.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:50:33.400073+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:50:38.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-22T20:50:38.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.2 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.2 2026-02-22T20:50:38.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:50:38.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:50:38.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:38.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.2 last_scrub_stamp 2026-02-22T20:50:38.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.2 2026-02-22T20:50:38.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:38.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:38.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.2") | .last_scrub_stamp' 2026-02-22T20:50:38.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:50:34.405812+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:50:38.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-22T20:50:38.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.3 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.3 2026-02-22T20:50:38.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:50:38.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:50:38.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:38.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.3 last_scrub_stamp 2026-02-22T20:50:38.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:50:38.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:38.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:38.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:50:38.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:50:35.430732+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:50:38.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-22T20:50:38.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.4 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.4 2026-02-22T20:50:38.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:50:38.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:50:38.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:38.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.4 last_scrub_stamp 2026-02-22T20:50:38.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.4 2026-02-22T20:50:38.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:38.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:38.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.4") | .last_scrub_stamp' 2026-02-22T20:50:38.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:50:36.429344+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:50:38.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-22T20:50:38.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.5 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.5 2026-02-22T20:50:38.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:50:38.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:50:38.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:38.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2026-02-22T20:50:38.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2026-02-22T20:50:38.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:38.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:38.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2026-02-22T20:50:38.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:32.520342+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:38.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:39.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:39.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:39.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2026-02-22T20:50:39.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2026-02-22T20:50:39.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:39.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:39.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2026-02-22T20:50:39.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:32.520342+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:39.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:40.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:40.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:40.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2026-02-22T20:50:40.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2026-02-22T20:50:40.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:40.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:40.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2026-02-22T20:50:41.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T20:49:32.520342+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:41.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:50:42.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:50:42.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:42.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.5 last_scrub_stamp 2026-02-22T20:50:42.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.5 2026-02-22T20:50:42.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:42.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:42.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.5") | .last_scrub_stamp' 2026-02-22T20:50:42.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:50:37.383862+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:42.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:50:42.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-22T20:50:42.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.6 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:42.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.6 2026-02-22T20:50:42.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:42.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:50:42.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:50:42.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:42.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.6 last_scrub_stamp 2026-02-22T20:50:42.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.6 2026-02-22T20:50:42.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:42.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:42.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.6") | .last_scrub_stamp' 2026-02-22T20:50:42.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:50:38.382443+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:42.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:50:42.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-22T20:50:42.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.7 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:42.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.7 2026-02-22T20:50:42.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:42.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:50:42.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:50:42.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:42.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.7 last_scrub_stamp 2026-02-22T20:50:42.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.7 2026-02-22T20:50:42.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:42.459 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:42.459 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.7") | .last_scrub_stamp' 2026-02-22T20:50:42.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:50:39.360251+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:42.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:50:42.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6415: TEST_request_scrub_priority: for opg in $otherpgs $pg 2026-02-22T20:50:42.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6417: TEST_request_scrub_priority: wait_for_scrub 1.1 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:42.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.1 2026-02-22T20:50:42.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:42.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:50:42.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:50:42.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:50:42.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.1 last_scrub_stamp 2026-02-22T20:50:42.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.1 2026-02-22T20:50:42.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:50:42.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:50:42.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.1") | .last_scrub_stamp' 2026-02-22T20:50:42.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:49:51.293624+0000 '>' 2026-02-22T20:49:27.173434+0000 2026-02-22T20:50:42.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:50:42.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: grep 'log_channel.*scrub ok' td/osd-scrub-repair/osd.0.log 2026-02-22T20:50:42.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: grep -v purged_snaps 2026-02-22T20:50:42.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: head -1 2026-02-22T20:50:42.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: sed 's/.*[[]DBG[]]//' 2026-02-22T20:50:42.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6421: TEST_request_scrub_priority: grep -q 1.1 2026-02-22T20:50:42.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:50:42.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:50:42.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:50:42.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:50:42.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:50:42.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:50:42.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:50:42.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:50:42.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:50:42.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:50:42.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:50:42.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:50:42.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:50:42.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:50:42.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:50:42.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:50:42.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:50:42.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:50:42.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:50:42.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:50:42.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:50:42.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:50:42.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:50:42.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:50:42.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:50:42.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:50:42.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:50:42.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:50:42.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:50:42.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:50:42.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:50:42.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:50:42.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:50:42.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:50:42.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:50:42.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:50:42.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:50:42.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:50:42.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:50:42.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:50:42.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:50:42.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:50:42.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:50:42.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:50:42.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:50:42.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:50:42.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:50:42.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:50:42.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:50:42.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:50:42.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:50:42.916 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:50:42.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:50:42.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:50:42.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:50:42.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:50:42.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:50:42.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:50:42.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:50:42.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:50:42.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:50:42.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:50:42.923 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:50:42.923 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:50:42.923 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:50:42.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:50:42.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:50:42.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:50:42.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:50:42.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:50:42.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_scrub_warning td/osd-scrub-repair 2026-02-22T20:50:42.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6011: TEST_scrub_warning: local dir=td/osd-scrub-repair 2026-02-22T20:50:42.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6012: TEST_scrub_warning: local poolname=psr_pool 2026-02-22T20:50:42.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6013: TEST_scrub_warning: local objname=POBJ 2026-02-22T20:50:42.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6014: TEST_scrub_warning: local scrubs=5 2026-02-22T20:50:42.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6015: TEST_scrub_warning: local deep_scrubs=5 2026-02-22T20:50:42.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6016: TEST_scrub_warning: local i1_day=86400 2026-02-22T20:50:42.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6017: TEST_scrub_warning: calc 86400 '*' 7 2026-02-22T20:50:42.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: calc: awk 'BEGIN{print 86400 * 7}' 2026-02-22T20:50:42.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6017: TEST_scrub_warning: local i7_days=604800 2026-02-22T20:50:42.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6018: TEST_scrub_warning: calc 86400 '*' 14 2026-02-22T20:50:42.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: calc: awk 'BEGIN{print 86400 * 14}' 2026-02-22T20:50:42.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6018: TEST_scrub_warning: local i14_days=1209600 2026-02-22T20:50:42.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6019: TEST_scrub_warning: local overdue=0.5 2026-02-22T20:50:42.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6020: TEST_scrub_warning: calc 604800 + 86400 + '(' 604800 '*' 0.5 ')' 2026-02-22T20:50:42.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: calc: awk 'BEGIN{print 604800 + 86400 + ( 604800 * 0.5 )}' 2026-02-22T20:50:42.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6020: TEST_scrub_warning: local conf_overdue_seconds=993600 2026-02-22T20:50:42.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6021: TEST_scrub_warning: calc 1209600 + 86400 + '(' 1209600 '*' 0.5 ')' 2026-02-22T20:50:42.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: calc: awk 'BEGIN{print 1209600 + 86400 + ( 1209600 * 0.5 )}' 2026-02-22T20:50:42.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6021: TEST_scrub_warning: local pool_overdue_seconds=1900800 2026-02-22T20:50:42.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6023: TEST_scrub_warning: run_mon td/osd-scrub-repair a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T20:50:42.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:50:42.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:50:42.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:50:42.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:50:42.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:50:42.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T20:50:42.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:50:42.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:50:42.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:50:42.959 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:50:42.959 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:50:42.959 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:50:42.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:50:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-22T20:50:42.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:50:42.989 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:50:42.989 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:50:42.989 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:50:42.989 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:50:42.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:50:42.990 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:50:42.991 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:50:42.991 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:50:42.991 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:50:42.991 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:50:42.991 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:50:42.991 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:50:42.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:50:42.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:50:43.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:50:43.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:50:43.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:50:43.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:50:43.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:50:43.048 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:50:43.049 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:50:43.049 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:50:43.049 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:50:43.049 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:50:43.049 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:50:43.049 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:50:43.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:50:43.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:50:43.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6024: TEST_scrub_warning: run_mgr td/osd-scrub-repair x --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2026-02-22T20:50:43.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:50:43.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:50:43.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:50:43.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:50:43.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:50:43.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:50:43.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:50:43.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:50:43.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:50:43.201 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:50:43.201 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:50:43.202 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:50:43.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:50:43.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:50:43.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_warn_pg_not_scrubbed_ratio=0.5 --mon_warn_pg_not_deep_scrubbed_ratio=0.5 2026-02-22T20:50:43.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6025: TEST_scrub_warning: run_osd td/osd-scrub-repair 0 --osd_scrub_backoff_ratio=0 2026-02-22T20:50:43.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:50:43.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:50:43.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:50:43.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:50:43.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:50:43.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:50:43.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:50:43.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:50:43.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:50:43.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:50:43.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:50:43.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:50:43.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:50:43.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:50:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:50:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:50:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:50:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:50:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:50:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_scrub_backoff_ratio=0 2026-02-22T20:50:43.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:50:43.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:50:43.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=eaa26822-addb-4fb4-8cc1-66cceb6ea743 2026-02-22T20:50:43.228 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 eaa26822-addb-4fb4-8cc1-66cceb6ea743 2026-02-22T20:50:43.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 eaa26822-addb-4fb4-8cc1-66cceb6ea743' 2026-02-22T20:50:43.229 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:50:43.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAjbJtpStTSDRAAI004kuKbeLKq1teCQOpq3g== 2026-02-22T20:50:43.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAjbJtpStTSDRAAI004kuKbeLKq1teCQOpq3g=="}' 2026-02-22T20:50:43.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new eaa26822-addb-4fb4-8cc1-66cceb6ea743 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:50:43.327 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:50:43.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:50:43.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 --mkfs --key AQAjbJtpStTSDRAAI004kuKbeLKq1teCQOpq3g== --osd-uuid eaa26822-addb-4fb4-8cc1-66cceb6ea743 2026-02-22T20:50:43.352 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:50:43.343+0000 7f0c7a254a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:50:43.353 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:50:43.343+0000 7f0c7a254a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:50:43.354 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:50:43.347+0000 7f0c7a254a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:50:43.355 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:50:43.347+0000 7f0c7a254a40 -1 bdev(0x557ac8b66c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:50:43.355 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:50:43.347+0000 7f0c7a254a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:50:44.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:50:44.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:50:44.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:50:44.135 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:50:44.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:50:44.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:50:44.339 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:50:44.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_backoff_ratio=0 2026-02-22T20:50:44.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:50:44.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:50:44.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:50:44.363 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:50:44.355+0000 7f9698a12a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:50:44.365 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:50:44.355+0000 7f9698a12a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:50:44.366 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:50:44.355+0000 7f9698a12a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:50:44.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:50:44.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:50:44.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:50:44.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:50:44.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:50:44.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:50:44.497 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:50:44.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:50:44.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:50:44.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:50:44.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:50:44.772 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:50:44.763+0000 7f9698a12a40 -1 Falling back to public interface 2026-02-22T20:50:45.306 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:50:45.295+0000 7f9698a12a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:50:45.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:50:45.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:50:45.661 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:50:45.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:50:45.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:50:45.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:50:45.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:50:46.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:50:46.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:50:46.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:50:46.828 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:50:46.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:50:46.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:50:46.987 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1321376811,v1:127.0.0.1:6803/1321376811] [v2:127.0.0.1:6804/1321376811,v1:127.0.0.1:6805/1321376811] exists,up eaa26822-addb-4fb4-8cc1-66cceb6ea743 2026-02-22T20:50:46.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:50:46.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:50:46.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:50:46.989 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: expr 5 + 5 2026-02-22T20:50:46.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: seq 1 10 2026-02-22T20:50:46.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:50:46.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-1 1 1 2026-02-22T20:50:46.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-1 1 1 2026-02-22T20:50:47.194 INFO:tasks.workunit.client.0.trial150.stderr:pool 'psr_pool-1' created 2026-02-22T20:50:47.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:50:48.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-22T20:50:48.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:50:48.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:50:48.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:50:48.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:50:48.207 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:50:48.207 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:50:48.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:50:48.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:50:48.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:50:48.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:50:48.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:50:48.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:50:48.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:50:48.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:50:48.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:50:48.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:50:48.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:50:48.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:50:48.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:50:48.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836482 2026-02-22T20:50:48.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836482 2026-02-22T20:50:48.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836482' 2026-02-22T20:50:48.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:50:48.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836482 2026-02-22T20:50:48.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:50:48.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:50:48.560 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836482 2026-02-22T20:50:48.560 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:50:48.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836482 2026-02-22T20:50:48.562 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836482 2026-02-22T20:50:48.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2026-02-22T20:50:48.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:50:48.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836482 2026-02-22T20:50:48.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:50:49.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:50:49.720 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:50:49.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836482 2026-02-22T20:50:49.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:50:49.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:50:49.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:50:50.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:50:50.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:50:50.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:50:50.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:50:50.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:50:50.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:50:50.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:50:50.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:50:50.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:50:50.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:50:50.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:50:50.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:50:50.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:50:50.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:50:50.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:50:50.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 1 = 1 ']' 2026-02-22T20:50:50.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6033: TEST_scrub_warning: ceph osd pool set psr_pool-1 scrub_max_interval 1209600 2026-02-22T20:50:50.629 INFO:tasks.workunit.client.0.trial150.stderr:set pool 1 scrub_max_interval to 1209600 2026-02-22T20:50:50.642 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:50:50.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 1 = 6 ']' 2026-02-22T20:50:50.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:50:50.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-2 1 1 2026-02-22T20:50:50.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-2 1 1 2026-02-22T20:50:50.834 INFO:tasks.workunit.client.0.trial150.stderr:pool 'psr_pool-2' created 2026-02-22T20:50:50.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:50:51.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-22T20:50:51.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:50:51.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:50:51.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:50:51.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:50:51.847 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:50:51.847 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:50:51.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:50:51.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:50:51.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:50:51.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:50:51.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:50:51.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:50:51.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:50:51.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:50:51.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:50:52.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:50:52.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:50:52.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:50:52.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:50:52.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-22T20:50:52.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-22T20:50:52.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T20:50:52.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:50:52.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-22T20:50:52.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:50:52.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:50:52.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-22T20:50:52.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:50:52.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-22T20:50:52.205 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836484 2026-02-22T20:50:52.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T20:50:52.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:50:52.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:50:52.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:50:53.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:50:53.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:50:53.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:50:53.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:50:54.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:50:54.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:50:54.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-22T20:50:54.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:50:54.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:50:54.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:50:54.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 2 == 0 2026-02-22T20:50:54.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:50:54.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:50:54.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:50:54.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:50:54.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:50:54.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:50:54.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:50:55.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=2 2026-02-22T20:50:55.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:50:55.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:50:55.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:50:55.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 2 = 2 2026-02-22T20:50:55.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:50:55.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:50:55.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 2 = 1 ']' 2026-02-22T20:50:55.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:50:55.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 2 = 6 ']' 2026-02-22T20:50:55.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:50:55.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-3 1 1 2026-02-22T20:50:55.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-3 1 1 2026-02-22T20:50:55.456 INFO:tasks.workunit.client.0.trial150.stderr:pool 'psr_pool-3' created 2026-02-22T20:50:55.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:50:56.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-22T20:50:56.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:50:56.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:50:56.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:50:56.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:50:56.469 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:50:56.470 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:50:56.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:50:56.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:50:56.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:50:56.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:50:56.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:50:56.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:50:56.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:50:56.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:50:56.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:50:56.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:50:56.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:50:56.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:50:56.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:50:56.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-22T20:50:56.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-22T20:50:56.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-22T20:50:56.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:50:56.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-22T20:50:56.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:50:56.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:50:56.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-22T20:50:56.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:50:56.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-22T20:50:56.827 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836486 2026-02-22T20:50:56.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-22T20:50:56.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:50:56.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836486 2026-02-22T20:50:56.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:50:57.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:50:57.987 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:50:58.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-22T20:50:58.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:50:58.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:50:58.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:50:58.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 3 == 0 2026-02-22T20:50:58.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:50:58.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:50:58.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:50:58.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:50:58.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:50:58.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:50:58.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:50:58.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=3 2026-02-22T20:50:58.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:50:58.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:50:58.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:50:58.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 3 = 3 2026-02-22T20:50:58.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:50:58.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:50:58.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 3 = 1 ']' 2026-02-22T20:50:58.686 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:50:58.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 3 = 6 ']' 2026-02-22T20:50:58.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:50:58.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-4 1 1 2026-02-22T20:50:58.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-4 1 1 2026-02-22T20:50:58.891 INFO:tasks.workunit.client.0.trial150.stderr:pool 'psr_pool-4' created 2026-02-22T20:50:58.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:50:59.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-22T20:50:59.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:50:59.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:50:59.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:50:59.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:50:59.906 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:50:59.906 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:50:59.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:50:59.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:50:59.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:51:00.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:51:00.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:51:00.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:51:00.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:51:00.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:51:00.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:51:00.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:51:00.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:51:00.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:51:00.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:51:00.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-22T20:51:00.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-22T20:51:00.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-22T20:51:00.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:51:00.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-22T20:51:00.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:51:00.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:51:00.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-22T20:51:00.263 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:51:00.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-22T20:51:00.265 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836487 2026-02-22T20:51:00.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-22T20:51:00.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:00.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T20:51:00.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:01.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:51:01.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:01.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T20:51:01.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:02.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:51:02.590 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:02.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836487 2026-02-22T20:51:02.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:51:02.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:02.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:02.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:51:02.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:51:02.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:51:02.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:51:02.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:51:02.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:51:02.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:51:02.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:51:03.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:51:03.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:51:03.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:03.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:03.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:51:03.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:51:03.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:51:03.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 4 = 1 ']' 2026-02-22T20:51:03.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:03.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 4 = 6 ']' 2026-02-22T20:51:03.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:51:03.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-5 1 1 2026-02-22T20:51:03.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-5 1 1 2026-02-22T20:51:03.478 INFO:tasks.workunit.client.0.trial150.stderr:pool 'psr_pool-5' created 2026-02-22T20:51:03.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:51:04.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-22T20:51:04.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:51:04.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:51:04.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:51:04.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:51:04.492 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:51:04.492 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:51:04.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:51:04.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:51:04.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:51:04.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:51:04.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:51:04.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:51:04.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:51:04.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:51:04.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:51:04.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:51:04.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:51:04.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:51:04.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:51:04.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-22T20:51:04.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-22T20:51:04.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-22T20:51:04.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:51:04.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-22T20:51:04.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:51:04.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:51:04.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-22T20:51:04.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:51:04.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-22T20:51:04.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-22T20:51:04.857 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836489 2026-02-22T20:51:04.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:05.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836489 2026-02-22T20:51:05.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:06.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:51:06.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:06.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-22T20:51:06.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:51:06.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:06.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:06.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 5 == 0 2026-02-22T20:51:06.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:51:06.368 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:51:06.368 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:51:06.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:51:06.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:51:06.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:51:06.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:51:06.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=5 2026-02-22T20:51:06.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:51:06.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:06.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:06.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 5 = 5 2026-02-22T20:51:06.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:51:06.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:51:06.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 5 = 1 ']' 2026-02-22T20:51:06.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:06.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 5 = 6 ']' 2026-02-22T20:51:06.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:51:06.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-6 1 1 2026-02-22T20:51:06.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-6 1 1 2026-02-22T20:51:06.920 INFO:tasks.workunit.client.0.trial150.stderr:pool 'psr_pool-6' created 2026-02-22T20:51:06.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:51:07.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-22T20:51:07.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:51:07.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:51:07.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:51:07.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:51:07.935 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:51:07.935 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:51:07.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:51:07.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:51:07.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:51:08.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:51:08.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:51:08.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:51:08.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:51:08.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:51:08.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:51:08.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:51:08.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:51:08.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:51:08.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:51:08.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-22T20:51:08.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-22T20:51:08.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T20:51:08.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:51:08.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-22T20:51:08.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:51:08.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:51:08.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-22T20:51:08.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:51:08.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-22T20:51:08.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T20:51:08.298 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836491 2026-02-22T20:51:08.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:08.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T20:51:08.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:09.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:51:09.459 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:09.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T20:51:09.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:10.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:51:10.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:10.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-22T20:51:10.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:51:10.787 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:10.787 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:10.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 6 == 0 2026-02-22T20:51:10.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:51:10.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:51:10.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:51:10.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:51:10.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:51:10.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:51:10.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:51:11.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=6 2026-02-22T20:51:11.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:51:11.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:11.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:11.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 6 = 6 2026-02-22T20:51:11.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:51:11.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:51:11.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 6 = 1 ']' 2026-02-22T20:51:11.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:11.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 6 = 6 ']' 2026-02-22T20:51:11.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6037: TEST_scrub_warning: ceph osd pool set psr_pool-6 deep_scrub_interval 1209600 2026-02-22T20:51:11.536 INFO:tasks.workunit.client.0.trial150.stderr:set pool 6 deep_scrub_interval to 1209600 2026-02-22T20:51:11.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:51:11.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-7 1 1 2026-02-22T20:51:11.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-7 1 1 2026-02-22T20:51:11.742 INFO:tasks.workunit.client.0.trial150.stderr:pool 'psr_pool-7' created 2026-02-22T20:51:11.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:51:12.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-22T20:51:12.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:51:12.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:51:12.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:51:12.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:51:12.756 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:51:12.756 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:51:12.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:51:12.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:51:12.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:51:12.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:51:12.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:51:12.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:51:12.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:51:12.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:51:12.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:51:13.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:51:13.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:51:13.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:51:13.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:51:13.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836493 2026-02-22T20:51:13.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836493 2026-02-22T20:51:13.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836493' 2026-02-22T20:51:13.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:51:13.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836493 2026-02-22T20:51:13.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:51:13.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:51:13.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836493 2026-02-22T20:51:13.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:51:13.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836493 2026-02-22T20:51:13.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-22T20:51:13.115 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836493 2026-02-22T20:51:13.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:13.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836493 2026-02-22T20:51:13.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:14.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:51:14.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:14.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-22T20:51:14.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:51:14.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:14.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:14.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 7 == 0 2026-02-22T20:51:14.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:51:14.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:51:14.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:51:14.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:51:14.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:51:14.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:51:14.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:51:14.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=7 2026-02-22T20:51:14.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:51:14.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:14.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:14.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 7 = 7 2026-02-22T20:51:14.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:51:14.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:51:14.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 7 = 1 ']' 2026-02-22T20:51:14.986 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:14.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 7 = 6 ']' 2026-02-22T20:51:14.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:51:14.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-8 1 1 2026-02-22T20:51:14.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-8 1 1 2026-02-22T20:51:15.196 INFO:tasks.workunit.client.0.trial150.stderr:pool 'psr_pool-8' created 2026-02-22T20:51:15.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:51:16.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-22T20:51:16.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:51:16.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:51:16.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:51:16.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:51:16.210 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:51:16.210 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:51:16.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:51:16.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:51:16.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:51:16.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:51:16.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:51:16.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:51:16.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:51:16.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:51:16.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:51:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:51:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:51:16.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:51:16.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:51:16.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836495 2026-02-22T20:51:16.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836495 2026-02-22T20:51:16.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836495' 2026-02-22T20:51:16.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:51:16.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836495 2026-02-22T20:51:16.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:51:16.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:51:16.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836495 2026-02-22T20:51:16.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:51:16.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836495 2026-02-22T20:51:16.572 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836495 2026-02-22T20:51:16.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2026-02-22T20:51:16.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:16.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836495 2026-02-22T20:51:16.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:17.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:51:17.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:17.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836495 2026-02-22T20:51:17.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:51:17.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:17.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:18.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 8 == 0 2026-02-22T20:51:18.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:51:18.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:51:18.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:51:18.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:51:18.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:51:18.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:51:18.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:51:18.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=8 2026-02-22T20:51:18.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:51:18.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:18.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:18.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 = 8 2026-02-22T20:51:18.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:51:18.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:51:18.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 8 = 1 ']' 2026-02-22T20:51:18.449 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:18.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 8 = 6 ']' 2026-02-22T20:51:18.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:51:18.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-9 1 1 2026-02-22T20:51:18.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-9 1 1 2026-02-22T20:51:18.653 INFO:tasks.workunit.client.0.trial150.stderr:pool 'psr_pool-9' created 2026-02-22T20:51:18.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:51:19.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-22T20:51:19.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:51:19.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:51:19.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:51:19.666 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:51:19.667 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:51:19.667 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:51:19.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:51:19.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:51:19.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:51:19.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:51:19.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:51:19.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:51:19.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:51:19.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:51:19.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:51:19.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:51:19.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:51:19.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:51:19.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:51:20.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836496 2026-02-22T20:51:20.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836496 2026-02-22T20:51:20.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496' 2026-02-22T20:51:20.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:51:20.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836496 2026-02-22T20:51:20.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:51:20.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:51:20.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836496 2026-02-22T20:51:20.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:51:20.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836496 2026-02-22T20:51:20.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-22T20:51:20.030 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836496 2026-02-22T20:51:20.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:20.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836496 2026-02-22T20:51:20.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:21.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:51:21.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:21.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836496 2026-02-22T20:51:21.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:22.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:51:22.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:22.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836496 2026-02-22T20:51:22.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:51:22.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:22.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:22.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 9 == 0 2026-02-22T20:51:22.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:51:22.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:51:22.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:51:22.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:51:22.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:51:22.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:51:22.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:51:22.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=9 2026-02-22T20:51:22.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:51:22.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:22.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:23.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 9 = 9 2026-02-22T20:51:23.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:51:23.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:51:23.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 9 = 1 ']' 2026-02-22T20:51:23.057 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:23.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 9 = 6 ']' 2026-02-22T20:51:23.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6027: TEST_scrub_warning: for i in $(seq 1 $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:51:23.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6029: TEST_scrub_warning: create_pool psr_pool-10 1 1 2026-02-22T20:51:23.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create psr_pool-10 1 1 2026-02-22T20:51:23.242 INFO:tasks.workunit.client.0.trial150.stderr:pool 'psr_pool-10' created 2026-02-22T20:51:23.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:51:24.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6030: TEST_scrub_warning: wait_for_clean 2026-02-22T20:51:24.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:51:24.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:51:24.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:51:24.254 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:51:24.255 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:51:24.255 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:51:24.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:51:24.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:51:24.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:51:24.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:51:24.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:51:24.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:51:24.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:51:24.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:51:24.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:51:24.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:51:24.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:51:24.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:51:24.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:51:24.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836498 2026-02-22T20:51:24.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836498 2026-02-22T20:51:24.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836498' 2026-02-22T20:51:24.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:51:24.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836498 2026-02-22T20:51:24.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:51:24.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:51:24.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836498 2026-02-22T20:51:24.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:51:24.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836498 2026-02-22T20:51:24.613 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836498 2026-02-22T20:51:24.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2026-02-22T20:51:24.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:24.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836498 2026-02-22T20:51:24.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:25.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:51:25.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:25.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836498 2026-02-22T20:51:25.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:51:25.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:25.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:26.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 10 == 0 2026-02-22T20:51:26.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:51:26.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:51:26.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:51:26.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:51:26.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:51:26.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:51:26.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:51:26.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=10 2026-02-22T20:51:26.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:51:26.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:26.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:26.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 10 = 10 2026-02-22T20:51:26.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:51:26.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:51:26.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6031: TEST_scrub_warning: '[' 10 = 1 ']' 2026-02-22T20:51:26.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:26.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6035: TEST_scrub_warning: '[' 10 = 6 ']' 2026-02-22T20:51:26.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6042: TEST_scrub_warning: local primary=0 2026-02-22T20:51:26.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6044: TEST_scrub_warning: ceph osd set noscrub 2026-02-22T20:51:26.686 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:51:26.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6045: TEST_scrub_warning: ceph osd set nodeep-scrub 2026-02-22T20:51:26.891 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:51:26.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6046: TEST_scrub_warning: ceph config set global osd_scrub_interval_randomize_ratio 0 2026-02-22T20:51:27.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6047: TEST_scrub_warning: ceph config set global osd_deep_scrub_randomize_ratio 0 2026-02-22T20:51:27.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6048: TEST_scrub_warning: ceph config set global osd_scrub_max_interval 604800 2026-02-22T20:51:27.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6049: TEST_scrub_warning: ceph config set global osd_deep_scrub_interval 604800 2026-02-22T20:51:27.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: seq 1 5 2026-02-22T20:51:27.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-22T20:51:27.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 1 = 1 ']' 2026-02-22T20:51:27.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6056: TEST_scrub_warning: overdue_seconds=1900800 2026-02-22T20:51:27.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 1900800 + 100 2026-02-22T20:51:27.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 1.0 schedule-scrub 1900900 2026-02-22T20:51:27.656 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:51:27.656 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:51:27.656 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:51:27.656 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-01-31T20:49:47.650262+0000" 2026-02-22T20:51:27.656 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:51:27.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-22T20:51:27.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 2 = 1 ']' 2026-02-22T20:51:27.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6058: TEST_scrub_warning: overdue_seconds=993600 2026-02-22T20:51:27.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 993600 + 200 2026-02-22T20:51:27.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 2.0 schedule-scrub 993800 2026-02-22T20:51:27.731 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:51:27.731 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:51:27.731 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:51:27.731 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T08:48:07.725943+0000" 2026-02-22T20:51:27.731 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:51:27.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-22T20:51:27.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 3 = 1 ']' 2026-02-22T20:51:27.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6058: TEST_scrub_warning: overdue_seconds=993600 2026-02-22T20:51:27.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 993600 + 300 2026-02-22T20:51:27.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 3.0 schedule-scrub 993900 2026-02-22T20:51:27.808 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:51:27.808 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:51:27.808 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:51:27.808 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T08:46:27.802059+0000" 2026-02-22T20:51:27.808 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:51:27.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-22T20:51:27.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 4 = 1 ']' 2026-02-22T20:51:27.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6058: TEST_scrub_warning: overdue_seconds=993600 2026-02-22T20:51:27.816 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 993600 + 400 2026-02-22T20:51:27.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 4.0 schedule-scrub 994000 2026-02-22T20:51:27.882 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:51:27.882 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:51:27.882 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:51:27.882 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T08:44:47.876730+0000" 2026-02-22T20:51:27.883 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:51:27.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6052: TEST_scrub_warning: for i in $(seq 1 $scrubs) 2026-02-22T20:51:27.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6054: TEST_scrub_warning: '[' 5 = 1 ']' 2026-02-22T20:51:27.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6058: TEST_scrub_warning: overdue_seconds=993600 2026-02-22T20:51:27.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: expr 993600 + 500 2026-02-22T20:51:27.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6060: TEST_scrub_warning: ceph tell 5.0 schedule-scrub 994100 2026-02-22T20:51:27.958 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:51:27.958 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:51:27.958 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:51:27.958 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T08:43:07.951923+0000" 2026-02-22T20:51:27.958 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:51:27.966 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:27.967 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: expr 5 + 5 2026-02-22T20:51:27.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: seq 6 10 2026-02-22T20:51:27.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:51:27.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:27.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 6 = 6 ']' 2026-02-22T20:51:27.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6067: TEST_scrub_warning: overdue_seconds=1900800 2026-02-22T20:51:27.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 1900800 + 600 2026-02-22T20:51:27.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 6.0 schedule-deep-scrub 1901400 2026-02-22T20:51:28.041 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:51:28.041 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:51:28.041 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:51:28.041 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-01-31T20:41:28.034912+0000" 2026-02-22T20:51:28.042 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:51:28.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:51:28.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:28.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 7 = 6 ']' 2026-02-22T20:51:28.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6069: TEST_scrub_warning: overdue_seconds=993600 2026-02-22T20:51:28.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 993600 + 700 2026-02-22T20:51:28.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 7.0 schedule-deep-scrub 994300 2026-02-22T20:51:28.118 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:51:28.118 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:51:28.118 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:51:28.118 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T08:39:48.112109+0000" 2026-02-22T20:51:28.119 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:51:28.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:51:28.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:28.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 8 = 6 ']' 2026-02-22T20:51:28.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6069: TEST_scrub_warning: overdue_seconds=993600 2026-02-22T20:51:28.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 993600 + 800 2026-02-22T20:51:28.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 8.0 schedule-deep-scrub 994400 2026-02-22T20:51:28.195 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:51:28.195 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:51:28.195 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:51:28.195 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T08:38:08.189086+0000" 2026-02-22T20:51:28.195 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:51:28.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:51:28.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:28.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 9 = 6 ']' 2026-02-22T20:51:28.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6069: TEST_scrub_warning: overdue_seconds=993600 2026-02-22T20:51:28.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 993600 + 900 2026-02-22T20:51:28.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 9.0 schedule-deep-scrub 994500 2026-02-22T20:51:28.274 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:51:28.274 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:51:28.274 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:51:28.274 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T08:36:28.268497+0000" 2026-02-22T20:51:28.274 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:51:28.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6063: TEST_scrub_warning: for i in $(seq $(expr $scrubs + 1) $(expr $scrubs + $deep_scrubs)) 2026-02-22T20:51:28.282 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: expr 5 + 1 2026-02-22T20:51:28.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6065: TEST_scrub_warning: '[' 10 = 6 ']' 2026-02-22T20:51:28.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6069: TEST_scrub_warning: overdue_seconds=993600 2026-02-22T20:51:28.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: expr 993600 + 1000 2026-02-22T20:51:28.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6071: TEST_scrub_warning: ceph tell 10.0 schedule-deep-scrub 994600 2026-02-22T20:51:28.350 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:51:28.350 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:51:28.350 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:51:28.350 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T08:34:48.344230+0000" 2026-02-22T20:51:28.350 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:51:28.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6073: TEST_scrub_warning: flush_pg_stats 2026-02-22T20:51:28.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:51:28.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:51:28.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:51:28.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:51:28.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:51:28.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:51:28.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836500 2026-02-22T20:51:28.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836500 2026-02-22T20:51:28.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836500' 2026-02-22T20:51:28.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:51:28.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836500 2026-02-22T20:51:28.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:51:28.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:51:28.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836500 2026-02-22T20:51:28.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:51:28.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836500 2026-02-22T20:51:28.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2026-02-22T20:51:28.594 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836500 2026-02-22T20:51:28.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:28.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836499 -lt 21474836500 2026-02-22T20:51:28.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:29.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:51:29.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:29.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836500 2026-02-22T20:51:29.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6075: TEST_scrub_warning: ceph health 2026-02-22T20:51:30.083 INFO:tasks.workunit.client.0.trial150.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 5 pgs not scrubbed in time; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2026-02-22T20:51:30.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6076: TEST_scrub_warning: ceph health detail 2026-02-22T20:51:30.277 INFO:tasks.workunit.client.0.trial150.stdout:HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 5 pgs not deep-scrubbed in time; 5 pgs not scrubbed in time; 10 pool(s) have no replicas configured; OSD count 1 < osd_pool_default_size 3 2026-02-22T20:51:30.277 INFO:tasks.workunit.client.0.trial150.stdout:[WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2026-02-22T20:51:30.277 INFO:tasks.workunit.client.0.trial150.stdout:[WRN] PG_NOT_DEEP_SCRUBBED: 5 pgs not deep-scrubbed in time 2026-02-22T20:51:30.277 INFO:tasks.workunit.client.0.trial150.stdout: pg 10.0 not deep-scrubbed since 2026-02-11T08:34:48.344230+0000 2026-02-22T20:51:30.278 INFO:tasks.workunit.client.0.trial150.stdout: pg 9.0 not deep-scrubbed since 2026-02-11T08:36:28.268497+0000 2026-02-22T20:51:30.278 INFO:tasks.workunit.client.0.trial150.stdout: pg 8.0 not deep-scrubbed since 2026-02-11T08:38:08.189086+0000 2026-02-22T20:51:30.278 INFO:tasks.workunit.client.0.trial150.stdout: pg 7.0 not deep-scrubbed since 2026-02-11T08:39:48.112109+0000 2026-02-22T20:51:30.278 INFO:tasks.workunit.client.0.trial150.stdout: pg 6.0 not deep-scrubbed since 2026-01-31T20:41:28.034912+0000 2026-02-22T20:51:30.278 INFO:tasks.workunit.client.0.trial150.stdout:[WRN] PG_NOT_SCRUBBED: 5 pgs not scrubbed in time 2026-02-22T20:51:30.278 INFO:tasks.workunit.client.0.trial150.stdout: pg 5.0 not scrubbed since 2026-02-11T08:43:07.951923+0000 2026-02-22T20:51:30.278 INFO:tasks.workunit.client.0.trial150.stdout: pg 4.0 not scrubbed since 2026-02-11T08:44:47.876730+0000 2026-02-22T20:51:30.278 INFO:tasks.workunit.client.0.trial150.stdout: pg 3.0 not scrubbed since 2026-02-11T08:46:27.802059+0000 2026-02-22T20:51:30.278 INFO:tasks.workunit.client.0.trial150.stdout: pg 2.0 not scrubbed since 2026-02-11T08:48:07.725943+0000 2026-02-22T20:51:30.278 INFO:tasks.workunit.client.0.trial150.stdout: pg 1.0 not scrubbed since 2026-01-31T20:49:47.650262+0000 2026-02-22T20:51:30.278 INFO:tasks.workunit.client.0.trial150.stdout:[WRN] POOL_NO_REDUNDANCY: 10 pool(s) have no replicas configured 2026-02-22T20:51:30.279 INFO:tasks.workunit.client.0.trial150.stdout: pool 'psr_pool-1' has no replicas configured 2026-02-22T20:51:30.279 INFO:tasks.workunit.client.0.trial150.stdout: pool 'psr_pool-2' has no replicas configured 2026-02-22T20:51:30.279 INFO:tasks.workunit.client.0.trial150.stdout: pool 'psr_pool-3' has no replicas configured 2026-02-22T20:51:30.279 INFO:tasks.workunit.client.0.trial150.stdout: pool 'psr_pool-4' has no replicas configured 2026-02-22T20:51:30.279 INFO:tasks.workunit.client.0.trial150.stdout: pool 'psr_pool-5' has no replicas configured 2026-02-22T20:51:30.279 INFO:tasks.workunit.client.0.trial150.stdout: pool 'psr_pool-6' has no replicas configured 2026-02-22T20:51:30.279 INFO:tasks.workunit.client.0.trial150.stdout: pool 'psr_pool-7' has no replicas configured 2026-02-22T20:51:30.279 INFO:tasks.workunit.client.0.trial150.stdout: pool 'psr_pool-8' has no replicas configured 2026-02-22T20:51:30.279 INFO:tasks.workunit.client.0.trial150.stdout: pool 'psr_pool-9' has no replicas configured 2026-02-22T20:51:30.279 INFO:tasks.workunit.client.0.trial150.stdout: pool 'psr_pool-10' has no replicas configured 2026-02-22T20:51:30.280 INFO:tasks.workunit.client.0.trial150.stdout:[WRN] TOO_FEW_OSDS: OSD count 1 < osd_pool_default_size 3 2026-02-22T20:51:30.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6077: TEST_scrub_warning: ceph health 2026-02-22T20:51:30.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6077: TEST_scrub_warning: grep -q ' pgs not deep-scrubbed in time' 2026-02-22T20:51:30.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6078: TEST_scrub_warning: ceph health 2026-02-22T20:51:30.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6078: TEST_scrub_warning: grep -q ' pgs not scrubbed in time' 2026-02-22T20:51:30.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6083: TEST_scrub_warning: ceph health detail 2026-02-22T20:51:30.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6083: TEST_scrub_warning: grep 'not scrubbed since' 2026-02-22T20:51:30.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6083: TEST_scrub_warning: wc -l 2026-02-22T20:51:30.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6083: TEST_scrub_warning: COUNT=5 2026-02-22T20:51:30.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6085: TEST_scrub_warning: expr 5+5 2026-02-22T20:51:30.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6085: TEST_scrub_warning: (( 5 != 5 && 5 != 5+5 )) 2026-02-22T20:51:30.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6089: TEST_scrub_warning: ceph health detail 2026-02-22T20:51:30.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6089: TEST_scrub_warning: grep 'not deep-scrubbed since' 2026-02-22T20:51:30.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6089: TEST_scrub_warning: wc -l 2026-02-22T20:51:31.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6089: TEST_scrub_warning: COUNT=5 2026-02-22T20:51:31.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:6090: TEST_scrub_warning: '[' 5 '!=' 5 ']' 2026-02-22T20:51:31.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:51:31.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:51:31.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:51:31.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:51:31.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:51:31.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:51:31.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:51:31.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:51:31.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:51:31.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:51:31.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:51:31.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:51:31.167 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:51:31.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:51:31.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:51:31.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:51:31.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:51:31.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:51:31.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:51:31.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:51:31.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:51:31.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:51:31.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:51:31.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:51:31.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:51:31.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:51:31.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:51:31.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:51:31.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:51:31.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:51:31.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:51:31.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:51:31.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:51:31.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:51:31.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:51:31.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:51:31.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:51:31.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:51:31.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:51:31.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:51:31.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:51:31.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:51:31.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:51:31.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:51:31.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:51:31.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:51:31.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:51:31.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:51:31.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:51:31.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:51:31.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:51:31.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:51:31.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:51:31.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:51:31.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:51:31.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:51:31.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:51:31.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:51:31.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:51:31.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:51:31.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:51:31.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:51:31.200 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:51:31.200 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:51:31.200 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:51:31.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:51:31.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:51:31.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:51:31.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:51:31.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:51:31.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_skip_non_repair_during_recovery td/osd-scrub-repair 2026-02-22T20:51:31.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:160: TEST_skip_non_repair_during_recovery: local dir=td/osd-scrub-repair 2026-02-22T20:51:31.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:161: TEST_skip_non_repair_during_recovery: cluster_conf=(['osds_num']='2' ['pgs_in_pool']='4' ['pool_name']='nopool' ['pool_default_size']='2' ['extras']='--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true') 2026-02-22T20:51:31.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:161: TEST_skip_non_repair_during_recovery: local -A cluster_conf 2026-02-22T20:51:31.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:169: TEST_skip_non_repair_during_recovery: standard_scrub_cluster td/osd-scrub-repair cluster_conf 2026-02-22T20:51:31.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-repair 2026-02-22T20:51:31.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-22T20:51:31.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=2 2026-02-22T20:51:31.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2026-02-22T20:51:31.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=nopool 2026-02-22T20:51:31.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=nopool 2026-02-22T20:51:31.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=2 2026-02-22T20:51:31.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=2 2026-02-22T20:51:31.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local 'extra_pars=--osd_scrub_during_recovery=false --osd_debug_pretend_recovery_active=true' 2026-02-22T20:51:31.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-22T20:51:31.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-22T20:51:31.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-22T20:51:31.498 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 57903660-3dd4-4a2a-8c26-ab0204b04799 2026-02-22T20:51:31.600 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:51:31.631 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:31.620+0000 7f09a21caa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:31.632 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:31.620+0000 7f09a21caa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:31.633 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:31.624+0000 7f09a21caa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:31.634 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:31.624+0000 7f09a21caa40 -1 bdev(0x55db27210c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:51:31.634 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:31.624+0000 7f09a21caa40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:51:32.402 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:51:32.608 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:51:32.628 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:32.616+0000 7fee114cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:32.630 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:32.620+0000 7fee114cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:32.630 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:32.620+0000 7fee114cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:32.770 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:51:33.023 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:33.012+0000 7fee114cba40 -1 Falling back to public interface 2026-02-22T20:51:33.639 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:33.628+0000 7fee114cba40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:51:33.938 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:51:35.099 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:51:35.259 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3870174065,v1:127.0.0.1:6803/3870174065] [v2:127.0.0.1:6804/3870174065,v1:127.0.0.1:6805/3870174065] exists,up 57903660-3dd4-4a2a-8c26-ab0204b04799 2026-02-22T20:51:35.266 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 6d8902f2-047d-4f91-ab83-5d48cb7bd9a3 2026-02-22T20:51:35.443 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:51:35.475 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:35.464+0000 7fbac658ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:35.476 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:35.464+0000 7fbac658ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:35.477 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:35.468+0000 7fbac658ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:35.478 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:35.468+0000 7fbac658ca40 -1 bdev(0x55dd4bd74c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:51:35.478 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:35.468+0000 7fbac658ca40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:51:36.157 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:51:36.359 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:51:36.392 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:36.380+0000 7f5a2ccb7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:36.394 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:36.384+0000 7f5a2ccb7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:36.395 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:36.384+0000 7f5a2ccb7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:36.516 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:51:36.783 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:36.772+0000 7f5a2ccb7a40 -1 Falling back to public interface 2026-02-22T20:51:37.416 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:37.404+0000 7f5a2ccb7a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:51:37.677 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:51:38.843 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:51:39.002 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1010175357,v1:127.0.0.1:6811/1010175357] [v2:127.0.0.1:6812/1010175357,v1:127.0.0.1:6813/1010175357] exists,up 6d8902f2-047d-4f91-ab83-5d48cb7bd9a3 2026-02-22T20:51:40.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:170: TEST_skip_non_repair_during_recovery: local poolname=rbd 2026-02-22T20:51:40.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:171: TEST_skip_non_repair_during_recovery: create_rbd_pool 2026-02-22T20:51:40.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-22T20:51:40.154 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' does not exist 2026-02-22T20:51:40.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-22T20:51:40.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-22T20:51:40.386 INFO:tasks.workunit.client.0.trial150.stderr:pool 'rbd' created 2026-02-22T20:51:40.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:51:41.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-22T20:51:41.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:172: TEST_skip_non_repair_during_recovery: wait_for_clean 2026-02-22T20:51:41.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:51:41.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:51:41.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:51:41.680 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:51:41.680 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:51:41.681 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:51:41.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:51:41.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:51:41.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:51:41.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:51:41.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:51:41.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:51:41.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:51:41.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:51:41.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:51:41.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:51:41.963 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:51:41.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:51:41.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:51:41.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:51:42.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836489 2026-02-22T20:51:42.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836489 2026-02-22T20:51:42.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489' 2026-02-22T20:51:42.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:51:42.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:51:42.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-22T20:51:42.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-22T20:51:42.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836489 1-38654705669' 2026-02-22T20:51:42.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:51:42.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836489 2026-02-22T20:51:42.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:51:42.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:51:42.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836489 2026-02-22T20:51:42.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:51:42.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836489 2026-02-22T20:51:42.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-22T20:51:42.117 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836489 2026-02-22T20:51:42.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:42.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836489 2026-02-22T20:51:42.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:43.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:51:43.287 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:43.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836488 -lt 21474836489 2026-02-22T20:51:43.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:44.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:51:44.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:44.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836489 2026-02-22T20:51:44.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:51:44.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-22T20:51:44.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:51:44.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:51:44.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-22T20:51:44.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:51:44.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-22T20:51:44.619 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705669 2026-02-22T20:51:44.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-22T20:51:44.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:51:44.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705671 -lt 38654705669 2026-02-22T20:51:44.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:51:44.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:44.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:44.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:51:44.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:51:44.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:51:44.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:51:44.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:51:44.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:51:44.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:51:44.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:51:45.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:51:45.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:51:45.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:45.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:45.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:51:45.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:51:45.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:51:45.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:174: TEST_skip_non_repair_during_recovery: add_something td/osd-scrub-repair rbd 2026-02-22T20:51:45.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:51:45.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=rbd 2026-02-22T20:51:45.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-22T20:51:45.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:51:45.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:51:45.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:51:45.519 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:51:45.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:51:45.724 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:51:45.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:51:45.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:51:45.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool rbd put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:51:45.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:175: TEST_skip_non_repair_during_recovery: get_not_primary rbd SOMETHING 2026-02-22T20:51:45.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-22T20:51:45.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=SOMETHING 2026-02-22T20:51:45.763 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd SOMETHING 2026-02-22T20:51:45.763 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-22T20:51:45.763 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:51:45.763 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd SOMETHING 2026-02-22T20:51:45.763 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:51:45.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:51:45.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd SOMETHING 2026-02-22T20:51:45.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:51:46.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:175: TEST_skip_non_repair_during_recovery: scrub_and_not_schedule td/osd-scrub-repair rbd 0 2026-02-22T20:51:46.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:214: scrub_and_not_schedule: local dir=td/osd-scrub-repair 2026-02-22T20:51:46.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:215: scrub_and_not_schedule: local poolname=rbd 2026-02-22T20:51:46.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:216: scrub_and_not_schedule: local osd=0 2026-02-22T20:51:46.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:221: scrub_and_not_schedule: get_pg rbd SOMETHING 2026-02-22T20:51:46.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-22T20:51:46.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:51:46.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd SOMETHING 2026-02-22T20:51:46.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:51:46.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:221: scrub_and_not_schedule: local pg=1.3 2026-02-22T20:51:46.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:222: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2026-02-22T20:51:46.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:51:46.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:51:46.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:51:46.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:51:46.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:222: scrub_and_not_schedule: local last_scrub=2026-02-22T20:51:40.379542+0000 2026-02-22T20:51:46.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:223: scrub_and_not_schedule: ceph tell 1.3 schedule-scrub 2026-02-22T20:51:46.475 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:51:46.476 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:51:46.476 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:51:46.476 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-20T20:51:46.469947+0000" 2026-02-22T20:51:46.476 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:51:46.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:228: scrub_and_not_schedule: sleep 3 2026-02-22T20:51:49.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i=0 )) 2026-02-22T20:51:49.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i < 3 )) 2026-02-22T20:51:49.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2026-02-22T20:51:49.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:51:49.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:51:49.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:51:49.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:51:49.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: test 2026-02-20T20:51:46.469947+0000 '>' 2026-02-22T20:51:40.379542+0000 2026-02-22T20:51:49.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: scrub_and_not_schedule: sleep 1 2026-02-22T20:51:50.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i++ )) 2026-02-22T20:51:50.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i < 3 )) 2026-02-22T20:51:50.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2026-02-22T20:51:50.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:51:50.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:51:50.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:51:50.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:51:50.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: test 2026-02-20T20:51:46.469947+0000 '>' 2026-02-22T20:51:40.379542+0000 2026-02-22T20:51:50.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: scrub_and_not_schedule: sleep 1 2026-02-22T20:51:51.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i++ )) 2026-02-22T20:51:51.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i < 3 )) 2026-02-22T20:51:51.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: get_last_scrub_stamp 1.3 2026-02-22T20:51:51.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.3 2026-02-22T20:51:51.813 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:51:51.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:51:51.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.3") | .last_scrub_stamp' 2026-02-22T20:51:51.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:230: scrub_and_not_schedule: test 2026-02-20T20:51:46.469947+0000 '>' 2026-02-22T20:51:40.379542+0000 2026-02-22T20:51:51.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:233: scrub_and_not_schedule: sleep 1 2026-02-22T20:51:52.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i++ )) 2026-02-22T20:51:52.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:229: scrub_and_not_schedule: (( i < 3 )) 2026-02-22T20:51:52.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:239: scrub_and_not_schedule: objectstore_tool td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:51:52.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:51:52.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:51:52.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:51:53.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:51:53.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING list-attrs 2026-02-22T20:51:53.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:51:53.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:51:53.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:51:53.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:51:53.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:51:53.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING list-attrs 2026-02-22T20:51:53.360 INFO:tasks.workunit.client.0.trial150.stdout:_ 2026-02-22T20:51:53.360 INFO:tasks.workunit.client.0.trial150.stdout:snapset 2026-02-22T20:51:53.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:51:53.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:51:53.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:51:53.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:51:53.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:51:53.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:51:53.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:51:53.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:51:53.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:51:53.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:51:53.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:51:53.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:51:53.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:51:53.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:51:53.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:51:53.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:51:53.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:51:53.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:51:53.450 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:51:53.450 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:51:53.450 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:51:53.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:51:53.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:51:53.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:51:53.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:51:53.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:51:53.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:51:53.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:51:53.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:51:53.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:51:53.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:51:53.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:51:53.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:51:53.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:51:53.451 INFO:tasks.workunit.client.0.trial150.stderr:start osd.0 2026-02-22T20:51:53.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:51:53.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:51:53.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:51:53.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:51:53.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:51:53.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:51:53.469 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:53.456+0000 7faab355ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:53.470 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:53.460+0000 7faab355ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:53.471 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:53.460+0000 7faab355ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:53.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:51:53.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:51:53.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:51:53.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:51:53.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:51:53.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:51:53.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:51:53.611 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:51:53.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:51:53.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:51:53.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:51:53.791 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:53.780+0000 7faab355ca40 -1 Falling back to public interface 2026-02-22T20:51:54.498 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:54.488+0000 7faab355ca40 -1 osd.0 19 log_to_monitors true 2026-02-22T20:51:54.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:51:54.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:51:54.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:51:54.770 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:51:54.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:51:54.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:51:54.930 INFO:tasks.workunit.client.0.trial150.stderr:osd.0 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [5,19) [v2:127.0.0.1:6802/2050311995,v1:127.0.0.1:6803/2050311995] [v2:127.0.0.1:6804/2050311995,v1:127.0.0.1:6805/2050311995] exists,up 57903660-3dd4-4a2a-8c26-ab0204b04799 2026-02-22T20:51:54.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:51:54.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:51:54.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:51:54.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:51:54.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:51:54.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:51:54.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:51:54.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:51:54.931 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:51:54.931 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:51:54.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:51:54.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:51:54.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:51:55.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:51:55.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:51:55.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:51:55.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:51:55.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:51:55.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:51:55.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:51:55.207 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:51:55.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:51:55.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:51:55.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:51:55.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-22T20:51:55.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-22T20:51:55.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810' 2026-02-22T20:51:55.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:51:55.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:51:55.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705683 2026-02-22T20:51:55.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705683 2026-02-22T20:51:55.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-98784247810 1-38654705683' 2026-02-22T20:51:55.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:51:55.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-98784247810 2026-02-22T20:51:55.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:51:55.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:51:55.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-98784247810 2026-02-22T20:51:55.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:51:55.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-22T20:51:55.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 98784247810' 2026-02-22T20:51:55.361 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 98784247810 2026-02-22T20:51:55.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:55.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 98784247810 2026-02-22T20:51:55.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:51:56.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:51:56.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:51:56.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247811 -lt 98784247810 2026-02-22T20:51:56.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:51:56.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705683 2026-02-22T20:51:56.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:51:56.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:51:56.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705683 2026-02-22T20:51:56.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:51:56.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705683 2026-02-22T20:51:56.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705683' 2026-02-22T20:51:56.698 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 38654705683 2026-02-22T20:51:56.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:51:56.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705684 -lt 38654705683 2026-02-22T20:51:56.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:51:56.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:56.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:57.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-22T20:51:57.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:51:57.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:51:57.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:51:57.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:51:57.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:51:57.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:51:57.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:51:57.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-22T20:51:57.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:51:57.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:51:57.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:51:57.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-22T20:51:57.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:51:57.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:51:57.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:240: scrub_and_not_schedule: rados --pool rbd get SOMETHING td/osd-scrub-repair/COPY 2026-02-22T20:51:57.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:241: scrub_and_not_schedule: diff td/osd-scrub-repair/ORIGINAL td/osd-scrub-repair/COPY 2026-02-22T20:51:57.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:51:57.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:51:57.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:51:57.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:51:57.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:51:57.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:51:57.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:51:57.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:51:57.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:51:57.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:51:57.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:51:57.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:51:57.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:51:57.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:51:57.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:51:57.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:51:57.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:51:57.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:51:57.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:51:57.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:51:57.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:51:57.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:51:57.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:51:57.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:51:57.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:51:57.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:51:57.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:51:57.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:51:57.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:51:57.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:51:57.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:51:57.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:51:57.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:51:57.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:51:57.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:51:57.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:51:57.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:51:57.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:51:57.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:51:57.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:51:57.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:51:57.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:51:57.556 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:51:57.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:51:57.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:51:57.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:51:57.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:51:57.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:51:57.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:51:57.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:51:57.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:51:57.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:51:57.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:51:57.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:51:57.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:51:57.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:51:57.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:51:57.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:51:57.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:51:57.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:51:57.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:51:57.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:51:57.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:51:57.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:51:57.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:51:57.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:51:57.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:51:57.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:51:57.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:51:57.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:51:57.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_appends td/osd-scrub-repair 2026-02-22T20:51:57.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:962: TEST_unfound_erasure_coded_appends: unfound_erasure_coded td/osd-scrub-repair false 2026-02-22T20:51:57.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:911: unfound_erasure_coded: local dir=td/osd-scrub-repair 2026-02-22T20:51:57.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:912: unfound_erasure_coded: local allow_overwrites=false 2026-02-22T20:51:57.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:913: unfound_erasure_coded: local poolname=ecpool 2026-02-22T20:51:57.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:914: unfound_erasure_coded: local payload=ABCDEF 2026-02-22T20:51:57.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-22T20:51:57.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:51:57.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:51:57.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:51:57.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:51:57.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:51:57.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:51:57.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:51:57.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:51:57.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:51:57.594 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:51:57.594 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:51:57.594 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:51:57.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:51:57.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:51:57.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:51:57.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:51:57.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:51:57.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:51:57.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:51:57.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:51:57.626 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:51:57.626 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:51:57.627 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:51:57.627 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:51:57.627 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:51:57.627 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:51:57.627 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:51:57.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:51:57.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:51:57.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:51:57.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:51:57.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:51:57.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:51:57.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:51:57.685 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:51:57.685 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:51:57.685 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:51:57.685 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:51:57.685 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:51:57.685 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:51:57.685 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:51:57.686 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:51:57.686 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:51:57.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-22T20:51:57.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:51:57.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:51:57.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:51:57.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:51:57.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:51:57.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:51:57.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:51:57.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:51:57.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:51:57.835 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:51:57.835 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:51:57.835 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:51:57.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:51:57.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:51:57.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:51:57.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: seq 0 3 2026-02-22T20:51:57.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-22T20:51:57.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2026-02-22T20:51:57.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:51:57.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:51:57.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:51:57.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:51:57.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:51:57.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:51:57.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:51:57.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:51:57.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:51:57.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:51:57.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:51:57.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:51:57.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:51:57.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:51:57.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:51:57.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:51:57.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:51:57.861 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:51:57.861 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:51:57.861 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:51:57.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:51:57.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:51:57.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:51:57.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:51:57.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:51:57.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:51:57.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:51:57.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:51:57.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:51:57.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:51:57.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:51:57.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:51:57.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:51:57.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:51:57.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:51:57.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=104b55f6-b3ce-46a4-a891-ae4f435233c4 2026-02-22T20:51:57.864 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 104b55f6-b3ce-46a4-a891-ae4f435233c4 2026-02-22T20:51:57.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 104b55f6-b3ce-46a4-a891-ae4f435233c4' 2026-02-22T20:51:57.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:51:57.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBtbJtp4Fq7MxAARG13xCjUWV7kE4YYPU7GNA== 2026-02-22T20:51:57.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBtbJtp4Fq7MxAARG13xCjUWV7kE4YYPU7GNA=="}' 2026-02-22T20:51:57.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 104b55f6-b3ce-46a4-a891-ae4f435233c4 -i td/osd-scrub-repair/0/new.json 2026-02-22T20:51:57.964 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:51:57.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:51:57.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtbJtp4Fq7MxAARG13xCjUWV7kE4YYPU7GNA== --osd-uuid 104b55f6-b3ce-46a4-a891-ae4f435233c4 2026-02-22T20:51:57.989 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:57.976+0000 7f24fac43a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:57.990 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:57.980+0000 7f24fac43a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:57.991 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:57.980+0000 7f24fac43a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:57.991 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:57.980+0000 7f24fac43a40 -1 bdev(0x5601748b8c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:51:57.992 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:57.980+0000 7f24fac43a40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:51:58.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:51:58.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:51:58.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:51:58.826 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:51:58.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:51:59.027 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:51:59.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:51:59.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:51:59.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:51:59.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:51:59.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:51:59.044 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:59.032+0000 7f9220010a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:59.046 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:59.036+0000 7f9220010a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:59.046 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:59.036+0000 7f9220010a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:51:59.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:51:59.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:51:59.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:51:59.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:51:59.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:51:59.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:51:59.188 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:51:59.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:51:59.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:51:59.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:51:59.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:51:59.455 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:59.444+0000 7f9220010a40 -1 Falling back to public interface 2026-02-22T20:51:59.984 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:51:59.972+0000 7f9220010a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:52:00.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:00.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:00.348 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:52:00.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:00.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:00.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:52:00.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:01.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:01.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:01.513 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:52:01.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:52:01.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:01.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:52:01.669 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/91975386,v1:127.0.0.1:6803/91975386] [v2:127.0.0.1:6804/91975386,v1:127.0.0.1:6805/91975386] exists,up 104b55f6-b3ce-46a4-a891-ae4f435233c4 2026-02-22T20:52:01.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:01.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:01.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:01.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-22T20:52:01.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2026-02-22T20:52:01.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:01.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:52:01.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:52:01.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:52:01.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:52:01.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:01.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:01.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:01.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:01.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:52:01.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:52:01.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:01.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:52:01.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:01.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:01.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:01.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:01.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:01.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:01.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:52:01.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:52:01.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:52:01.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:52:01.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d3cce019-7850-4311-9dfb-3c9420a8a4d2 2026-02-22T20:52:01.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d3cce019-7850-4311-9dfb-3c9420a8a4d2' 2026-02-22T20:52:01.677 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 d3cce019-7850-4311-9dfb-3c9420a8a4d2 2026-02-22T20:52:01.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:52:01.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBxbJtp4ZqfKBAAm2EI07fIL2sbuxhEa8WWGg== 2026-02-22T20:52:01.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBxbJtp4ZqfKBAAm2EI07fIL2sbuxhEa8WWGg=="}' 2026-02-22T20:52:01.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d3cce019-7850-4311-9dfb-3c9420a8a4d2 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:52:01.845 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:52:01.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:52:01.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxbJtp4ZqfKBAAm2EI07fIL2sbuxhEa8WWGg== --osd-uuid d3cce019-7850-4311-9dfb-3c9420a8a4d2 2026-02-22T20:52:01.872 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:01.860+0000 7fa4996c6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:01.874 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:01.864+0000 7fa4996c6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:01.875 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:01.864+0000 7fa4996c6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:01.875 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:01.864+0000 7fa4996c6a40 -1 bdev(0x55ab3f532c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:52:01.875 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:01.864+0000 7fa4996c6a40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:52:02.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:52:02.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:52:02.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:52:02.522 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:52:02.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:52:02.719 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:52:02.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:52:02.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:02.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:52:02.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:52:02.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:52:02.743 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:02.732+0000 7fa060870a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:02.745 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:02.736+0000 7fa060870a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:02.746 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:02.736+0000 7fa060870a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:02.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:52:02.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:02.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:52:02.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:02.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:02.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:02.880 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:52:02.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:02.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:02.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:52:03.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:03.146 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:03.136+0000 7fa060870a40 -1 Falling back to public interface 2026-02-22T20:52:03.807 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:03.796+0000 7fa060870a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:52:04.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:04.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:04.036 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:52:04.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:04.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:04.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:52:04.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:05.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:05.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:05.192 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:52:05.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:52:05.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:05.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:52:05.352 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2634064402,v1:127.0.0.1:6811/2634064402] [v2:127.0.0.1:6812/2634064402,v1:127.0.0.1:6813/2634064402] exists,up d3cce019-7850-4311-9dfb-3c9420a8a4d2 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:52:05.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:05.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:52:05.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:05.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:05.354 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:05.354 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:05.354 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:05.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:52:05.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:52:05.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:52:05.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fd175603-9d92-45e2-9fa5-a1cf6b5c273b 2026-02-22T20:52:05.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 fd175603-9d92-45e2-9fa5-a1cf6b5c273b' 2026-02-22T20:52:05.358 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 fd175603-9d92-45e2-9fa5-a1cf6b5c273b 2026-02-22T20:52:05.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:52:05.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB1bJtpvuS4FRAAC/iK9J0B1ZJqUz4Zu18Mrw== 2026-02-22T20:52:05.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB1bJtpvuS4FRAAC/iK9J0B1ZJqUz4Zu18Mrw=="}' 2026-02-22T20:52:05.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fd175603-9d92-45e2-9fa5-a1cf6b5c273b -i td/osd-scrub-repair/2/new.json 2026-02-22T20:52:05.533 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:52:05.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:52:05.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB1bJtpvuS4FRAAC/iK9J0B1ZJqUz4Zu18Mrw== --osd-uuid fd175603-9d92-45e2-9fa5-a1cf6b5c273b 2026-02-22T20:52:05.563 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:05.552+0000 7fbce965da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:05.565 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:05.552+0000 7fbce965da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:05.566 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:05.556+0000 7fbce965da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:05.566 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:05.556+0000 7fbce965da40 -1 bdev(0x5596a1674c00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:52:05.566 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:05.556+0000 7fbce965da40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:52:06.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:52:06.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:52:06.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:52:06.201 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:52:06.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:52:06.397 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:52:06.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:52:06.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:52:06.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:06.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:52:06.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:52:06.415 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:06.404+0000 7f7d5aff8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:06.417 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:06.404+0000 7f7d5aff8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:06.418 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:06.408+0000 7f7d5aff8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:06.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:52:06.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:06.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:52:06.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:06.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:06.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:06.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:06.558 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:52:06.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:06.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:52:06.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:06.843 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:06.832+0000 7f7d5aff8a40 -1 Falling back to public interface 2026-02-22T20:52:07.446 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:07.436+0000 7f7d5aff8a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:52:07.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:07.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:07.720 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:52:07.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:07.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:07.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:52:07.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:08.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:08.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:08.879 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:52:08.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:52:08.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:08.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:52:09.038 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1316737744,v1:127.0.0.1:6819/1316737744] [v2:127.0.0.1:6820/1316737744,v1:127.0.0.1:6821/1316737744] exists,up fd175603-9d92-45e2-9fa5-a1cf6b5c273b 2026-02-22T20:52:09.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:09.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:09.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:09.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-22T20:52:09.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2026-02-22T20:52:09.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:09.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:52:09.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-22T20:52:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:52:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:52:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:52:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:52:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:52:09.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:09.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:52:09.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:09.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:09.044 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:09.044 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:09.044 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:09.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:09.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:09.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:09.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:52:09.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:52:09.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:52:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:52:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:52:09.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:52:09.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=286f59e7-5517-4e1d-b106-20378c042dec 2026-02-22T20:52:09.046 INFO:tasks.workunit.client.0.trial150.stdout:add osd3 286f59e7-5517-4e1d-b106-20378c042dec 2026-02-22T20:52:09.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 286f59e7-5517-4e1d-b106-20378c042dec' 2026-02-22T20:52:09.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:52:09.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB5bJtpfp0BAxAAooWs89VU6P9lqXzoGkQ0/w== 2026-02-22T20:52:09.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB5bJtpfp0BAxAAooWs89VU6P9lqXzoGkQ0/w=="}' 2026-02-22T20:52:09.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 286f59e7-5517-4e1d-b106-20378c042dec -i td/osd-scrub-repair/3/new.json 2026-02-22T20:52:09.218 INFO:tasks.workunit.client.0.trial150.stdout:3 2026-02-22T20:52:09.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-22T20:52:09.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5bJtpfp0BAxAAooWs89VU6P9lqXzoGkQ0/w== --osd-uuid 286f59e7-5517-4e1d-b106-20378c042dec 2026-02-22T20:52:09.245 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:09.236+0000 7f431615da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:09.247 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:09.236+0000 7f431615da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:09.248 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:09.236+0000 7f431615da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:09.248 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:09.236+0000 7f431615da40 -1 bdev(0x55718b324c00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-22T20:52:09.248 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:09.236+0000 7f431615da40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-22T20:52:09.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-22T20:52:09.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:52:09.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-22T20:52:09.901 INFO:tasks.workunit.client.0.trial150.stdout:adding osd3 key to auth repository 2026-02-22T20:52:09.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:52:10.101 INFO:tasks.workunit.client.0.trial150.stdout:start osd.3 2026-02-22T20:52:10.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-22T20:52:10.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:10.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:52:10.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:52:10.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:52:10.125 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:10.116+0000 7f43c4493a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:10.127 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:10.116+0000 7f43c4493a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:10.128 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:10.116+0000 7f43c4493a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:10.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-22T20:52:10.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:10.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:52:10.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:10.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:10.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:10.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:10.264 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:52:10.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:10.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:52:10.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:10.515 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:10.504+0000 7f43c4493a40 -1 Falling back to public interface 2026-02-22T20:52:11.180 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:11.168+0000 7f43c4493a40 -1 osd.3 0 log_to_monitors true 2026-02-22T20:52:11.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:11.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:11.425 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:52:11.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:11.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:11.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:52:11.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:12.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:12.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:12.584 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:52:12.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:52:12.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:12.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:52:12.742 INFO:tasks.workunit.client.0.trial150.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3903487586,v1:127.0.0.1:6827/3903487586] [v2:127.0.0.1:6828/3903487586,v1:127.0.0.1:6829/3903487586] exists,up 286f59e7-5517-4e1d-b106-20378c042dec 2026-02-22T20:52:12.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:12.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:12.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:12.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: unfound_erasure_coded: create_ec_pool ecpool false k=2 m=2 2026-02-22T20:52:12.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-22T20:52:12.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:52:12.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=false 2026-02-22T20:52:12.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:52:12.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2026-02-22T20:52:12.986 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:52:12.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-22T20:52:12.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-22T20:52:13.290 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' created 2026-02-22T20:52:13.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:52:14.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' false = true ']' 2026-02-22T20:52:14.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:52:14.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:52:14.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:52:14.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:52:14.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:52:14.304 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:52:14.304 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:52:14.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:52:14.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:52:14.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:52:14.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:52:14.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:52:14.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:52:14.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:52:14.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:52:14.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:52:14.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:52:14.585 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:52:14.585 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:52:14.585 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:52:14.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:52:14.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:14.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:52:14.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-22T20:52:14.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-22T20:52:14.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T20:52:14.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:14.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:52:14.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-22T20:52:14.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-22T20:52:14.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963' 2026-02-22T20:52:14.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:14.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:52:14.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509443 2026-02-22T20:52:14.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509443 2026-02-22T20:52:14.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-64424509443' 2026-02-22T20:52:14.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:14.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:52:14.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378626 2026-02-22T20:52:14.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378626 2026-02-22T20:52:14.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672963 2-64424509443 3-81604378626' 2026-02-22T20:52:14.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:14.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-22T20:52:14.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:14.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:52:14.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-22T20:52:14.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:14.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-22T20:52:14.881 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836484 2026-02-22T20:52:14.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T20:52:14.882 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:15.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:52:15.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:52:16.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:52:16.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:16.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:52:16.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:52:17.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:52:17.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:17.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836484 2026-02-22T20:52:17.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:17.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-22T20:52:17.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:17.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:52:17.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-22T20:52:17.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:17.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-22T20:52:17.369 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672963 2026-02-22T20:52:17.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-22T20:52:17.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:52:17.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672963 2026-02-22T20:52:17.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:17.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509443 2026-02-22T20:52:17.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:17.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:52:17.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509443 2026-02-22T20:52:17.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:17.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509443 2026-02-22T20:52:17.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2026-02-22T20:52:17.536 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509443 2026-02-22T20:52:17.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:52:17.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509443 -lt 64424509443 2026-02-22T20:52:17.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:17.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378626 2026-02-22T20:52:17.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:17.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:52:17.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-81604378626 2026-02-22T20:52:17.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:17.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=81604378626 2026-02-22T20:52:17.709 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.3 seq 81604378626 2026-02-22T20:52:17.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 81604378626' 2026-02-22T20:52:17.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:52:17.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 81604378626 -lt 81604378626 2026-02-22T20:52:17.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:52:17.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:17.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:18.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:52:18.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:18.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:18.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:18.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:18.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:18.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:18.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:18.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:52:18.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:18.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:18.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:52:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:52:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:52:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:52:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:924: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-22T20:52:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:52:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:52:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-22T20:52:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:52:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:52:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:52:18.575 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:52:18.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:52:18.776 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:52:18.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:52:18.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:52:18.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:52:18.808 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: unfound_erasure_coded: get_primary ecpool SOMETHING 2026-02-22T20:52:18.808 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-22T20:52:18.808 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:52:18.808 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:52:18.808 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:52:18.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: unfound_erasure_coded: local primary=1 2026-02-22T20:52:18.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: get_osds ecpool SOMETHING 2026-02-22T20:52:18.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: sed -e s/1// 2026-02-22T20:52:18.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-22T20:52:18.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-22T20:52:18.968 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:52:18.968 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:52:19.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:52:19.122 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:52:19.123 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:52:19.123 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:52:19.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 3 2026-02-22T20:52:19.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: osds=('0' '2' '3') 2026-02-22T20:52:19.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: local -a osds 2026-02-22T20:52:19.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: unfound_erasure_coded: local not_primary_first=0 2026-02-22T20:52:19.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:929: unfound_erasure_coded: local not_primary_second=2 2026-02-22T20:52:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:930: unfound_erasure_coded: local not_primary_third=3 2026-02-22T20:52:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: unfound_erasure_coded: pids= 2026-02-22T20:52:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:52:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:52:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:52:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 444773"' 2026-02-22T20:52:19.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 444773' 2026-02-22T20:52:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:937: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:52:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:52:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:52:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 444775"' 2026-02-22T20:52:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 444775' 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:938: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/444776: /' 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 444777"' 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 444777' 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: unfound_erasure_coded: wait_background pids 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 444773 444775 444777' 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:52:19.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 444773 2026-02-22T20:52:19.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/444779: /' 2026-02-22T20:52:19.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:52:19.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/444783: /' 2026-02-22T20:52:19.810 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:52:19.810 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:52:19.810 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:52:19.810 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:52:19.810 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:52:19.810 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:52:19.811 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:52:19.811 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:52:19.811 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:52:19.811 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:52:19.811 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:52:19.811 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:52:19.811 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:52:19.811 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:52:19.811 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:52:19.811 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:52:19.811 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:52:19.811 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:52:19.812 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:52:19.812 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:52:19.812 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:52:19.812 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:52:19.812 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-22T20:52:19.812 INFO:tasks.workunit.client.0.trial150.stderr:444783: remove 3#1:eb822e21:::SOMETHING:head# 2026-02-22T20:52:19.812 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:52:19.812 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:19.812 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:52:19.812 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:52:19.812 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:52:19.812 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:52:19.813 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:19.813 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:19.813 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: 444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:52:19.813 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:52:19.813 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T20:52:19.813 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:52:19.813 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:52:19.813 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:52:19.813 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:52:19.813 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T20:52:19.813 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:52:19.813 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:52:19.814 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2026-02-22T20:52:19.815 INFO:tasks.workunit.client.0.trial150.stderr:444779: remove 2#1:eb822e21:::SOMETHING:head# 2026-02-22T20:52:19.815 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-22T20:52:19.815 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:19.815 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:52:19.815 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:52:19.815 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:52:19.815 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:52:19.815 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:19.815 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:19.816 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:19.816 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:19.816 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:52:19.816 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:52:19.816 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:19.816 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:52:19.816 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:19.817 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:52:19.817 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:19.817 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:19.817 INFO:tasks.workunit.client.0.trial150.stderr:444779: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:19.817 INFO:tasks.workunit.client.0.trial150.stderr:444779: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:19.817 INFO:tasks.workunit.client.0.trial150.stderr:444779: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:19.817 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:19.817 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:19.817 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:19.817 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:19.817 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:19.817 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:444779: start osd.2 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:4activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:52:19.818 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:52:19.819 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:19.819 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:52:19.819 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:19.819 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:52:19.819 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:19.819 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:19.819 INFO:tasks.workunit.client.0.trial150.stderr:444783: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:19.819 INFO:tasks.workunit.client.0.trial150.stderr:444783: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:19.819 INFO:tasks.workunit.client.0.trial150.stderr:444783: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:19.819 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:19.819 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:19.819 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:19.820 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:19.820 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:19.820 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:19.820 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:19.820 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:19.820 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:19.820 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:52:19.820 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:52:19.820 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:52:19.820 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:52:19.821 INFO:tasks.workunit.client.0.trial150.stderr:444783: start osd.3 2026-02-22T20:52:19.821 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:19.825 INFO:tasks.workunit.client.0.trial150.stderr:4444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:52:19.825 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:52:19.825 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:52:19.826 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:52:19.826 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:52:19.826 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:52:19.826 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:52:19.826 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:52:19.826 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:52:19.826 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:52:19.826 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:52:19.826 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:52:19.826 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:52:19.827 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:52:19.827 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:52:19.827 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:52:19.827 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:52:19.827 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:52:19.827 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:52:19.827 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:52:19.827 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:52:19.827 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:52:19.828 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-22T20:52:19.828 INFO:tasks.workunit.client.0.trial150.stderr:444776: remove 1#1:eb822e21:::SOMETHING:head# 2026-02-22T20:52:19.828 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:52:19.828 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:19.828 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:52:19.828 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:52:19.828 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:52:19.828 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:52:19.828 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:19.828 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:19.829 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:19.829 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:19.829 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:19.830 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:19.831 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:19.831 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:19.831 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:19.831 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:19.831 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:19.831 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:19.831 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:52:19.831 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:52:19.831 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:52:19.831 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:52:19.831 INFO:tasks.workunit.client.0.trial150.stderr:444776: start osd.0 2026-02-22T20:52:19.831 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:21.303 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:52:21.303 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:52:21.303 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:52:21.303 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:52:21.303 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:52:21.303 INFO:tasks.workunit.client.0.trial150.stderr:444783: 2026-02-22T20:52:19.820+0000 7f254727ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:21.303 INFO:tasks.workunit.client.0.trial150.stderr:444783: 2026-02-22T20:52:19.820+0000 7f254727ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: 2026-02-22T20:52:19.824+0000 7f254727ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: 0 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: 2026-02-22T20:52:20.332+0000 7f254727ea40 -1 Falling back to public interface 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: 2026-02-22T20:52:20.984+0000 7f254727ea40 -1 osd.3 26 log_to_monitors true 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: 1 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: osd.3 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [19,26) [v2:127.0.0.1:6826/3064677810,v1:127.0.0.1:6827/3064677810] [v2:127.0.0.1:6828/3064677810,v1:127.0.0.1:6829/3064677810] exists,up 286f59e7-5517-4e1d-b106-20378c042dec 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:52:21.304 INFO:tasks.workunit.client.0.trial150.stderr:444783: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:52:21.313 INFO:tasks.workunit.client.0.trial150.stderr:444783: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: ge44779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-22T20:52:21.313 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:52:21.313 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:52:21.313 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:52:21.313 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:52:21.313 INFO:tasks.workunit.client.0.trial150.stderr:444779: 2026-02-22T20:52:19.820+0000 7f75588f1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:21.313 INFO:tasks.workunit.client.0.trial150.stderr:444779: 2026-02-22T20:52:19.820+0000 7f75588f1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: 2026-02-22T20:52:19.824+0000 7f75588f1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: 0 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: 2026-02-22T20:52:20.312+0000 7f75588f1a40 -1 Falling back to public interface 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: 2026-02-22T20:52:20.964+0000 7f75588f1a40 -1 osd.2 26 log_to_monitors true 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: 1 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: osd.2 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [15,26) [v2:127.0.0.1:6818/4051519093,v1:127.0.0.1:6819/4051519093] [v2:127.0.0.1:6820/4051519093,v1:127.0.0.1:6821/4051519093] exists,up fd175603-9d92-45e2-9fa5-a1cf6b5c273b 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:52:21.314 INFO:tasks.workunit.client.0.trial150.stderr:444779: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:52:21.319 INFO:tasks.workunit.client.0.trial150.stderr:444779: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: ge44776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:52:21.319 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:52:21.319 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:52:21.319 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:52:21.319 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:52:21.319 INFO:tasks.workunit.client.0.trial150.stderr:444776: 2026-02-22T20:52:19.832+0000 7f252e025a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:21.319 INFO:tasks.workunit.client.0.trial150.stderr:444776: 2026-02-22T20:52:19.836+0000 7f252e025a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:21.319 INFO:tasks.workunit.client.0.trial150.stderr:444776: 2026-02-22T20:52:19.836+0000 7f252e025a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:21.320 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:52:21.320 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:21.320 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:52:21.320 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:21.320 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:21.320 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:21.320 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:21.320 INFO:tasks.workunit.client.0.trial150.stderr:444776: 0 2026-02-22T20:52:21.320 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:21.320 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:52:21.320 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:21.321 INFO:tasks.workunit.client.0.trial150.stderr:444776: 2026-02-22T20:52:20.292+0000 7f252e025a40 -1 Falling back to public interface 2026-02-22T20:52:21.321 INFO:tasks.workunit.client.0.trial150.stderr:444776: 2026-02-22T20:52:20.964+0000 7f252e025a40 -1 osd.0 26 log_to_monitors true 2026-02-22T20:52:21.321 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:21.321 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:21.321 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:21.321 INFO:tasks.workunit.client.0.trial150.stderr:444776: 1 2026-02-22T20:52:21.321 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:21.321 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:52:21.321 INFO:tasks.workunit.client.0.trial150.stderr:444776: osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [5,26) [v2:127.0.0.1:6802/1295366255,v1:127.0.0.1:6803/1295366255] [v2:127.0.0.1:6804/1295366255,v1:127.0.0.1:6805/1295366255] exists,up 104b55f6-b3ce-46a4-a891-ae4f435233c4 2026-02-22T20:52:21.321 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:21.321 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:21.322 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:21.322 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:52:21.322 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:52:21.322 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:52:21.322 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:52:21.322 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:52:21.322 INFO:tasks.workunit.client.0.trial150.stderr:444776: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:52:21.875 INFO:tasks.workunit.client.0.trial150.stderr:444776: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: gett_timeout_delays: echo true 2026-02-22T20:52:21.876 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:52:21.876 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:52:21.876 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:52:21.876 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:52:21.876 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:52:21.876 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:52:21.876 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:52:21.876 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:52:21.876 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:52:21.876 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:52:21.877 INFO:tasks.workunit.client.0.trial150.stderr:444783: 1 2026-02-22T20:52:21.877 INFO:tasks.workunit.client.0.trial150.stderr:444783: 2 2026-02-22T20:52:21.877 INFO:tasks.workunit.client.0.trial150.stderr:444783: 3' 2026-02-22T20:52:21.877 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:52:21.877 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:21.877 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:52:21.877 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-22T20:52:21.877 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-22T20:52:21.877 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882' 2026-02-22T20:52:21.877 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:21.877 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:52:21.877 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672966 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672966 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-42949672966' 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-42949672966 2-128849018882' 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-42949672966 2-128849018882 3-128849018882' 2026-02-22T20:52:21.878 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:21.879 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018882 2026-02-22T20:52:21.884 INFO:tasks.workunit.client.0.trial150.stderr:444783: //homet_timeout_delays: echo true 2026-02-22T20:52:21.884 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:52:21.884 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:52:21.885 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:52:21.885 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:52:21.885 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:52:21.885 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:52:21.885 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:52:21.885 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:52:21.885 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:52:21.885 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:52:21.885 INFO:tasks.workunit.client.0.trial150.stderr:444779: 1 2026-02-22T20:52:21.885 INFO:tasks.workunit.client.0.trial150.stderr:444779: 2 2026-02-22T20:52:21.885 INFO:tasks.workunit.client.0.trial150.stderr:444779: 3' 2026-02-22T20:52:21.885 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018884 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018884 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884' 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672968 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672968 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884 1-42949672968' 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:52:21.886 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018884 2026-02-22T20:52:21.887 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018884 2026-02-22T20:52:21.887 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884 1-42949672968 2-128849018884' 2026-02-22T20:52:21.887 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:21.887 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:52:21.887 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018883 2026-02-22T20:52:21.887 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018883 2026-02-22T20:52:21.887 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884 1-42949672968 2-128849018884 3-128849018883' 2026-02-22T20:52:21.887 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:21.887 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018884 2026-02-22T20:52:21.888 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home_timeout_delays: echo true 2026-02-22T20:52:21.888 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:52:21.888 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:52:21.888 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:52:21.888 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:52:21.888 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:52:21.888 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:52:21.888 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:52:21.888 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:52:21.889 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:52:21.889 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:52:21.889 INFO:tasks.workunit.client.0.trial150.stderr:444776: 1 2026-02-22T20:52:21.889 INFO:tasks.workunit.client.0.trial150.stderr:444776: 2 2026-02-22T20:52:21.889 INFO:tasks.workunit.client.0.trial150.stderr:444776: 3' 2026-02-22T20:52:21.889 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:52:21.889 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:21.889 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:52:21.889 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018883 2026-02-22T20:52:21.889 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018883 2026-02-22T20:52:21.889 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018883' 2026-02-22T20:52:21.889 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:21.890 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:52:21.890 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-22T20:52:21.890 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-22T20:52:21.890 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018883 1-42949672967' 2026-02-22T20:52:21.890 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:21.890 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:52:21.890 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018883 2026-02-22T20:52:21.890 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018883 2026-02-22T20:52:21.890 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018883 1-42949672967 2-128849018883' 2026-02-22T20:52:21.890 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:21.890 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:52:21.890 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018884 2026-02-22T20:52:21.891 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018884 2026-02-22T20:52:21.891 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018883 1-42949672967 2-128849018883 3-128849018884' 2026-02-22T20:52:21.891 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:21.891 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018883 2026-02-22T20:52:23.560 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home//ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:23.560 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:52:23.560 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018884 2026-02-22T20:52:23.560 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:23.560 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018884 2026-02-22T20:52:23.560 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018884' 2026-02-22T20:52:23.560 INFO:tasks.workunit.client.0.trial150.stderr:444779: waiting osd.0 seq 128849018884 2026-02-22T20:52:23.561 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:23.561 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 128849018884 2026-02-22T20:52:23.561 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:52:23.561 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:52:23.561 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:23.561 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018884 2026-02-22T20:52:23.561 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:23.562 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672968 2026-02-22T20:52:23.562 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:23.562 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:52:23.562 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672968 2026-02-22T20:52:23.562 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:23.562 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672968 2026-02-22T20:52:23.562 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2026-02-22T20:52:23.562 INFO:tasks.workunit.client.0.trial150.stderr:444779: waiting osd.1 seq 42949672968 2026-02-22T20:52:23.562 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:52:23.562 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672968 2026-02-22T20:52:23.563 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:23.563 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-128849018884 2026-02-22T20:52:23.563 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:23.563 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:52:23.563 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-128849018884 2026-02-22T20:52:23.563 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:23.563 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018884 2026-02-22T20:52:23.563 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 128849018884' 2026-02-22T20:52:23.563 INFO:tasks.workunit.client.0.trial150.stderr:444779: waiting osd.2 seq 128849018884 2026-02-22T20:52:23.563 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:52:23.563 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018884 2026-02-22T20:52:23.564 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:23.574 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.s/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:23.575 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:52:23.575 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018882 2026-02-22T20:52:23.575 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:23.575 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-22T20:52:23.575 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2026-02-22T20:52:23.576 INFO:tasks.workunit.client.0.trial150.stderr:444783: waiting osd.0 seq 128849018882 2026-02-22T20:52:23.576 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:23.576 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 128849018882 2026-02-22T20:52:23.576 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:52:23.576 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:52:23.577 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:23.577 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018882 2026-02-22T20:52:23.577 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:23.578 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672966 2026-02-22T20:52:23.578 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:23.578 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:52:23.578 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672966 2026-02-22T20:52:23.578 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:23.578 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672966 2026-02-22T20:52:23.579 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2026-02-22T20:52:23.579 INFO:tasks.workunit.client.0.trial150.stderr:444783: waiting osd.1 seq 42949672966 2026-02-22T20:52:23.579 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:52:23.579 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672966 2026-02-22T20:52:23.579 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:23.579 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-128849018882 2026-02-22T20:52:23.579 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:23.579 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:52:23.580 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-128849018882 2026-02-22T20:52:23.580 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:23.580 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-22T20:52:23.580 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 128849018882' 2026-02-22T20:52:23.580 INFO:tasks.workunit.client.0.trial150.stderr:444783: waiting osd.2 seq 128849018882 2026-02-22T20:52:23.580 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:52:23.580 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018882 2026-02-22T20:52:23.580 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:23.594 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.subuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:23.595 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:52:23.595 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018883 2026-02-22T20:52:23.595 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:23.595 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018883 2026-02-22T20:52:23.595 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018883' 2026-02-22T20:52:23.595 INFO:tasks.workunit.client.0.trial150.stderr:444776: waiting osd.0 seq 128849018883 2026-02-22T20:52:23.595 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:23.595 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 128849018883 2026-02-22T20:52:23.595 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:52:23.595 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:52:23.596 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:23.596 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018883 2026-02-22T20:52:23.596 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:23.596 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-22T20:52:23.596 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:23.596 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:52:23.596 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-22T20:52:23.596 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:23.596 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-22T20:52:23.596 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-22T20:52:23.596 INFO:tasks.workunit.client.0.trial150.stderr:444776: waiting osd.1 seq 42949672967 2026-02-22T20:52:23.596 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:52:23.597 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672967 2026-02-22T20:52:23.597 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:23.597 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-128849018883 2026-02-22T20:52:23.597 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:23.597 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:52:23.597 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-128849018883 2026-02-22T20:52:23.597 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:23.597 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018883 2026-02-22T20:52:23.597 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 128849018883' 2026-02-22T20:52:23.597 INFO:tasks.workunit.client.0.trial150.stderr:444776: waiting osd.2 seq 128849018883 2026-02-22T20:52:23.597 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:52:23.598 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018883 2026-02-22T20:52:23.598 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:24.408 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shh:2267: flush_pg_stats: echo 3-128849018883 2026-02-22T20:52:24.409 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:24.409 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:52:24.409 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-128849018883 2026-02-22T20:52:24.409 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:24.409 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018883 2026-02-22T20:52:24.409 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 128849018883' 2026-02-22T20:52:24.409 INFO:tasks.workunit.client.0.trial150.stderr:444779: waiting osd.3 seq 128849018883 2026-02-22T20:52:24.409 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:52:24.409 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018883 2026-02-22T20:52:24.409 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:52:24.409 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:24.410 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:24.410 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:52:24.410 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:24.410 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:24.410 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:24.410 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:24.410 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:24.410 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:24.410 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:24.410 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:52:24.410 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:24.410 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:24.411 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:24.411 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:52:24.411 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-22T20:52:24.411 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:52:24.411 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-22T20:52:24.411 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:52:24.411 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:52:24.411 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:52:24.411 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:24.411 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:24.414 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expreh:2267: flush_pg_stats: echo 3-128849018882 2026-02-22T20:52:24.414 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:24.414 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:52:24.414 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-128849018882 2026-02-22T20:52:24.414 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:24.415 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-22T20:52:24.415 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 128849018882' 2026-02-22T20:52:24.415 INFO:tasks.workunit.client.0.trial150.stderr:444783: waiting osd.3 seq 128849018882 2026-02-22T20:52:24.415 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:52:24.415 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018882 2026-02-22T20:52:24.415 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:52:24.415 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:24.415 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:24.415 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:52:24.415 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:24.415 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:24.415 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-22T20:52:24.416 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:52:24.417 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:52:24.417 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:52:24.417 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:24.417 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:24.451 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expre:2267: flush_pg_stats: echo 3-128849018884 2026-02-22T20:52:24.451 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:24.451 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:52:24.452 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-128849018884 2026-02-22T20:52:24.452 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:24.452 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018884 2026-02-22T20:52:24.452 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 128849018884' 2026-02-22T20:52:24.452 INFO:tasks.workunit.client.0.trial150.stderr:444776: waiting osd.3 seq 128849018884 2026-02-22T20:52:24.452 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:52:24.452 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018884 2026-02-22T20:52:24.452 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:52:24.452 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:24.452 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:24.452 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:52:24.452 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:24.453 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:24.453 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:24.453 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:24.453 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:24.453 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:24.453 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:24.453 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:52:24.453 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:24.453 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:24.453 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:24.453 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:52:24.453 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-22T20:52:24.454 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T20:52:24.454 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-22T20:52:24.454 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:52:24.454 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T20:52:24.454 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:52:24.454 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:24.454 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:25.156 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expresssion 2026-02-22T20:52:25.156 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:25.156 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:25.156 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:25.156 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:25.157 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:52:25.157 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:25.157 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:25.157 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:25.157 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:52:25.157 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:52:25.157 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:52:25.157 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:52:25.157 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:52:25.157 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:52:25.157 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:52:25.157 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:52:25.158 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:52:25.158 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:52:25.158 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:52:25.158 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:52:25.158 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:52:25.158 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:52:25.158 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:52:25.158 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:25.158 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:25.158 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:25.158 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:25.158 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:25.163 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length'ssion 2026-02-22T20:52:25.163 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:25.163 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:25.163 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:25.163 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:25.163 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:52:25.164 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:25.164 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:25.164 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:25.164 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:52:25.164 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:52:25.164 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:52:25.164 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:52:25.164 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:52:25.164 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:52:25.164 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:52:25.164 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:52:25.165 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:52:25.165 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:52:25.165 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:52:25.165 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:52:25.165 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:52:25.165 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:52:25.165 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:52:25.165 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:25.165 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:25.165 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:25.165 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:25.165 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:25.166 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:25.209 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_actision 2026-02-22T20:52:25.209 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:25.209 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:25.209 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:25.210 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:25.210 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:52:25.210 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:25.210 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:25.210 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:25.210 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:52:25.210 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:52:25.210 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:52:25.210 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:52:25.210 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:52:25.210 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:52:25.210 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:25.211 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:25.212 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:26.277 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_activve_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:26.277 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:52:26.277 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:26.278 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:26.278 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:26.278 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:52:26.278 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:52:26.278 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:52:26.278 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:52:26.278 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:52:26.278 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:52:26.278 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:52:26.278 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:52:26.278 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:26.279 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:26.280 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:52:26.280 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:26.280 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:26.280 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helper 2026-02-22T20:52:26.280 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:52:26.281 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:52:26.282 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:52:26.282 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:52:26.282 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-22T20:52:26.282 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:52:26.282 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-22T20:52:26.282 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:52:26.282 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:26.282 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:26.282 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:26.282 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:26.282 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:26.282 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:26.283 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:26.283 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:52:26.283 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:26.283 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:26.337 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpere_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:26.337 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T20:52:26.337 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:26.337 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:26.337 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:26.337 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:52:26.338 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:52:26.338 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:52:26.338 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:52:26.338 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:52:26.338 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:52:26.338 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:52:26.338 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:52:26.338 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:52:26.338 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:52:26.338 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:52:26.338 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-22T20:52:26.338 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:52:26.339 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-22T20:52:26.339 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:52:26.339 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:26.339 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:26.339 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:26.339 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:26.339 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:26.339 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:26.339 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:26.339 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:52:26.339 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:26.339 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:26.527 INFO:tasks.workunit.client.0.trial150.stderr:444776: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:26.527 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:52:26.527 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:52:26.527 INFO:tasks.workunit.client.0.trial150.stderr:444776: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:52:26.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:52:26.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:52:26.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 444775 2026-02-22T20:52:27.837 INFO:tasks.workunit.client.0.trial150.stderr:s.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:27.837 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:52:27.837 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:52:27.837 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:52:27.837 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:52:27.838 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:52:27.838 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:52:27.838 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:52:27.838 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:52:27.838 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:52:27.838 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:52:27.838 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:52:27.838 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-22T20:52:27.838 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:52:27.839 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-22T20:52:27.839 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:52:27.839 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:27.839 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:27.839 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:27.839 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:27.839 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:27.839 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:27.840 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:27.840 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:52:27.840 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:27.840 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:27.840 INFO:tasks.workunit.client.0.trial150.stderr:444779: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:27.840 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:52:27.840 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:52:27.840 INFO:tasks.workunit.client.0.trial150.stderr:444779: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:52:27.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 444777 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:s.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:27.841 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:444783: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:444783: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: unfound_erasure_coded: return_code=0 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:941: unfound_erasure_coded: '[' 0 -ne 0 ']' 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: unfound_erasure_coded: get_pg ecpool SOMETHING 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:52:27.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:52:27.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: unfound_erasure_coded: local pg=1.0 2026-02-22T20:52:27.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: unfound_erasure_coded: repair 1.0 2026-02-22T20:52:27.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-22T20:52:27.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-22T20:52:27.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:52:27.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:52:27.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:52:27.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:52:28.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:52:13.283516+0000 2026-02-22T20:52:28.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-22T20:52:28.306 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0s0 on osd.1 to repair 2026-02-22T20:52:28.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-22T20:52:13.283516+0000 2026-02-22T20:52:28.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:52:28.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:52:13.283516+0000 2026-02-22T20:52:28.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:52:28.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:52:28.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:52:28.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:52:28.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:52:28.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:52:28.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:52:28.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:52:28.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:52:13.283516+0000 '>' 2026-02-22T20:52:13.283516+0000 2026-02-22T20:52:28.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:52:29.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:52:29.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:52:29.475 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:52:29.475 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:52:29.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:52:29.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:52:29.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:52:29.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:52:13.283516+0000 '>' 2026-02-22T20:52:13.283516+0000 2026-02-22T20:52:29.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:52:30.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:52:30.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:52:30.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:52:30.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:52:30.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:52:30.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:52:30.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:52:30.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:52:28.844408+0000 '>' 2026-02-22T20:52:13.283516+0000 2026-02-22T20:52:30.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:52:30.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: unfound_erasure_coded: seq 1 60 2026-02-22T20:52:30.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: unfound_erasure_coded: for f in `seq 1 60` 2026-02-22T20:52:30.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: ceph -s 2026-02-22T20:52:30.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: grep '1/1 objects unfound' 2026-02-22T20:52:30.991 INFO:tasks.workunit.client.0.trial150.stdout: 1/1 objects unfound (100.000%) 2026-02-22T20:52:30.992 INFO:tasks.workunit.client.0.trial150.stdout: 1/1 objects unfound (100.000%) 2026-02-22T20:52:30.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: break 2026-02-22T20:52:30.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: unfound_erasure_coded: ceph -s 2026-02-22T20:52:30.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: unfound_erasure_coded: grep '4 up' 2026-02-22T20:52:31.184 INFO:tasks.workunit.client.0.trial150.stdout: osd: 4 osds: 4 up (since 10s), 4 in (since 21s) 2026-02-22T20:52:31.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: unfound_erasure_coded: ceph -s 2026-02-22T20:52:31.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: unfound_erasure_coded: grep '4 in' 2026-02-22T20:52:31.373 INFO:tasks.workunit.client.0.trial150.stdout: osd: 4 osds: 4 up (since 10s), 4 in (since 22s) 2026-02-22T20:52:31.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: unfound_erasure_coded: ceph -s 2026-02-22T20:52:31.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: unfound_erasure_coded: grep '1/1 objects unfound' 2026-02-22T20:52:31.564 INFO:tasks.workunit.client.0.trial150.stdout: 1/1 objects unfound (100.000%) 2026-02-22T20:52:31.564 INFO:tasks.workunit.client.0.trial150.stdout: 1/1 objects unfound (100.000%) 2026-02-22T20:52:31.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:52:31.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:52:31.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:52:31.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:52:31.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:52:31.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:52:31.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:52:31.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:52:31.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:52:31.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:52:31.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:52:31.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:52:31.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:52:31.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:52:31.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:52:31.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:52:31.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:52:31.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:52:31.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:52:31.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:52:31.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:52:31.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:52:31.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:52:31.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:52:31.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:31.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:31.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:52:31.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:52:31.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:52:31.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:63: run: for func in $funcs 2026-02-22T20:52:31.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:64: run: setup td/osd-scrub-repair 2026-02-22T20:52:31.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-repair 2026-02-22T20:52:31.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-repair 2026-02-22T20:52:31.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:52:31.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:52:31.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:52:31.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:52:31.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:52:31.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:52:31.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:52:31.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:52:31.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:52:31.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:52:31.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:52:31.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:52:31.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:52:31.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:52:31.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:52:31.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:52:31.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:52:31.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:52:31.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:52:31.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:52:31.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:52:31.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:52:31.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:52:31.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:31.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:31.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:52:31.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:52:31.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:52:31.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-repair 2026-02-22T20:52:31.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:52:31.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:31.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:31.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.40073 2026-02-22T20:52:31.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:52:31.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:52:31.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:52:31.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-repair 1' TERM HUP INT 2026-02-22T20:52:31.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:65: run: TEST_unfound_erasure_coded_overwrites td/osd-scrub-repair 2026-02-22T20:52:31.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:966: TEST_unfound_erasure_coded_overwrites: '[' true = true ']' 2026-02-22T20:52:31.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:967: TEST_unfound_erasure_coded_overwrites: unfound_erasure_coded td/osd-scrub-repair true 2026-02-22T20:52:31.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:911: unfound_erasure_coded: local dir=td/osd-scrub-repair 2026-02-22T20:52:31.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:912: unfound_erasure_coded: local allow_overwrites=true 2026-02-22T20:52:31.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:913: unfound_erasure_coded: local poolname=ecpool 2026-02-22T20:52:31.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:914: unfound_erasure_coded: local payload=ABCDEF 2026-02-22T20:52:31.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:916: unfound_erasure_coded: run_mon td/osd-scrub-repair a 2026-02-22T20:52:31.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-repair 2026-02-22T20:52:31.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:52:31.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:52:31.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:52:31.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-repair/a 2026-02-22T20:52:31.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-repair/a --run-dir=td/osd-scrub-repair 2026-02-22T20:52:31.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:52:31.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:31.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:31.951 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:31.951 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:31.951 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:31.951 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:31.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-repair/a '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-repair/log --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T20:52:31.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:52:31.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:52:31.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:52:31.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:52:31.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:52:31.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:52:31.982 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:52:31.982 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:52:31.982 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:52:31.983 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:52:31.984 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:31.984 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:31.984 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:52:31.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:52:31.984 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get fsid 2026-02-22T20:52:32.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:52:32.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:52:32.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:52:32.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:52:32.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:52:32.043 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:52:32.043 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:52:32.043 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:52:32.043 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:52:32.043 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:32.043 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:32.043 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.40073/ceph-mon.a.asok 2026-02-22T20:52:32.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:52:32.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.40073/ceph-mon.a.asok config get mon_host 2026-02-22T20:52:32.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:917: unfound_erasure_coded: run_mgr td/osd-scrub-repair x 2026-02-22T20:52:32.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-repair 2026-02-22T20:52:32.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:52:32.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:52:32.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:52:32.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-repair/x 2026-02-22T20:52:32.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:52:32.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:52:32.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:32.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:32.197 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:32.197 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:32.197 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:32.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:32.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:52:32.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-repair/x '--log-file=td/osd-scrub-repair/$name.log' '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --run-dir=td/osd-scrub-repair '--pid-file=td/osd-scrub-repair/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:52:32.218 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: seq 0 3 2026-02-22T20:52:32.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-22T20:52:32.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 0 2026-02-22T20:52:32.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:32.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:52:32.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:52:32.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:52:32.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:52:32.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:32.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:32.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:32.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:32.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:52:32.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:52:32.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:32.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:52:32.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:32.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:52:32.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:32.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:32.222 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:32.222 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:32.222 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:32.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:32.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:32.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:32.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:52:32.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:52:32.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:32.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:32.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:32.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:32.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:32.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:32.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:52:32.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:52:32.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:52:32.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:52:32.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a27e3098-92ce-4822-8e46-b46bec819d7d 2026-02-22T20:52:32.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a27e3098-92ce-4822-8e46-b46bec819d7d' 2026-02-22T20:52:32.226 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 a27e3098-92ce-4822-8e46-b46bec819d7d 2026-02-22T20:52:32.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:52:32.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCQbJtpHbOkDRAAP3qjiumLecDE07BC+dpssw== 2026-02-22T20:52:32.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCQbJtpHbOkDRAAP3qjiumLecDE07BC+dpssw=="}' 2026-02-22T20:52:32.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a27e3098-92ce-4822-8e46-b46bec819d7d -i td/osd-scrub-repair/0/new.json 2026-02-22T20:52:32.323 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:52:32.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/0/new.json 2026-02-22T20:52:32.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCQbJtpHbOkDRAAP3qjiumLecDE07BC+dpssw== --osd-uuid a27e3098-92ce-4822-8e46-b46bec819d7d 2026-02-22T20:52:32.348 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:32.336+0000 7f4a662bba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:32.350 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:32.340+0000 7f4a662bba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:32.351 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:32.340+0000 7f4a662bba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:32.351 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:32.340+0000 7f4a662bba40 -1 bdev(0x5588e6a76c00 td/osd-scrub-repair/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:52:32.352 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:32.340+0000 7f4a662bba40 -1 bluestore(td/osd-scrub-repair/0) _read_fsid unparsable uuid 2026-02-22T20:52:33.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/0/keyring 2026-02-22T20:52:33.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:52:33.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:52:33.125 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:52:33.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:52:33.330 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:52:33.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:52:33.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:33.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:52:33.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:52:33.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:52:33.355 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:33.344+0000 7fc759dc2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:33.357 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:33.348+0000 7fc759dc2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:33.358 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:33.348+0000 7fc759dc2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:33.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:52:33.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:33.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:52:33.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:33.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:33.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:33.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:33.491 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:52:33.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:33.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:52:33.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:33.715 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:33.704+0000 7fc759dc2a40 -1 Falling back to public interface 2026-02-22T20:52:34.408 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:34.396+0000 7fc759dc2a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:52:34.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:34.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:34.654 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:52:34.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:34.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:34.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:52:34.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:35.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:35.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:35.820 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:52:35.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:52:35.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:52:35.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:35.980 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2512234512,v1:127.0.0.1:6803/2512234512] [v2:127.0.0.1:6804/2512234512,v1:127.0.0.1:6805/2512234512] exists,up a27e3098-92ce-4822-8e46-b46bec819d7d 2026-02-22T20:52:35.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:35.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:35.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:35.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-22T20:52:35.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 1 2026-02-22T20:52:35.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:35.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:52:35.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:52:35.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:52:35.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/1 2026-02-22T20:52:35.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:35.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:35.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:35.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:35.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/1' 2026-02-22T20:52:35.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/1/journal' 2026-02-22T20:52:35.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:35.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:52:35.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:35.986 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:52:35.986 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:35.986 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:35.986 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:35.986 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:35.987 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:35.987 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:35.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:35.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:35.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:52:35.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:52:35.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:35.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:35.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:35.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:35.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:35.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:35.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:52:35.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:52:35.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/1 2026-02-22T20:52:35.988 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:52:35.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a00e2e3d-8c40-4911-a3a5-d21702717260 2026-02-22T20:52:35.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a00e2e3d-8c40-4911-a3a5-d21702717260' 2026-02-22T20:52:35.989 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 a00e2e3d-8c40-4911-a3a5-d21702717260 2026-02-22T20:52:35.989 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:52:35.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCTbJtpMtArOxAA5MDjRAH8WguKVmqV1AZ5Mg== 2026-02-22T20:52:35.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCTbJtpMtArOxAA5MDjRAH8WguKVmqV1AZ5Mg=="}' 2026-02-22T20:52:35.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a00e2e3d-8c40-4911-a3a5-d21702717260 -i td/osd-scrub-repair/1/new.json 2026-02-22T20:52:36.161 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:52:36.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/1/new.json 2026-02-22T20:52:36.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTbJtpMtArOxAA5MDjRAH8WguKVmqV1AZ5Mg== --osd-uuid a00e2e3d-8c40-4911-a3a5-d21702717260 2026-02-22T20:52:36.188 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:36.176+0000 7f414c28ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:36.189 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:36.180+0000 7f414c28ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:36.190 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:36.180+0000 7f414c28ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:36.191 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:36.180+0000 7f414c28ca40 -1 bdev(0x55aaa5a62c00 td/osd-scrub-repair/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:52:36.191 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:36.180+0000 7f414c28ca40 -1 bluestore(td/osd-scrub-repair/1) _read_fsid unparsable uuid 2026-02-22T20:52:36.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/1/keyring 2026-02-22T20:52:36.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:52:36.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:52:36.853 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:52:36.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:52:37.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:52:37.059 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:52:37.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/1 --osd-journal=td/osd-scrub-repair/1/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:37.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:52:37.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:52:37.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:52:37.075 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:37.064+0000 7f7d5b7dda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:37.077 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:37.068+0000 7f7d5b7dda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:37.078 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:37.068+0000 7f7d5b7dda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:37.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:52:37.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:37.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:52:37.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:37.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:37.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:37.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:37.217 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:52:37.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:37.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:52:37.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:37.487 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:37.476+0000 7f7d5b7dda40 -1 Falling back to public interface 2026-02-22T20:52:38.083 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:38.072+0000 7f7d5b7dda40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:52:38.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:38.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:38.381 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:52:38.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:38.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:38.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:52:38.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:39.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:39.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:39.544 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:52:39.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:52:39.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:39.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:52:39.704 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2946711712,v1:127.0.0.1:6811/2946711712] [v2:127.0.0.1:6812/2946711712,v1:127.0.0.1:6813/2946711712] exists,up a00e2e3d-8c40-4911-a3a5-d21702717260 2026-02-22T20:52:39.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:39.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:39.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:39.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-22T20:52:39.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 2 2026-02-22T20:52:39.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:39.705 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:39.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:39.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:39.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:39.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:52:39.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:52:39.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:39.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:39.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:39.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:39.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:39.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:39.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:52:39.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:52:39.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:52:39.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:52:39.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fa3ef831-3b27-4a0a-9f3d-15cf0a774b94 2026-02-22T20:52:39.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 fa3ef831-3b27-4a0a-9f3d-15cf0a774b94' 2026-02-22T20:52:39.709 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 fa3ef831-3b27-4a0a-9f3d-15cf0a774b94 2026-02-22T20:52:39.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:52:39.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCXbJtpQrToKhAA8UjgAOKQmbUbVu0X2VVAAw== 2026-02-22T20:52:39.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCXbJtpQrToKhAA8UjgAOKQmbUbVu0X2VVAAw=="}' 2026-02-22T20:52:39.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fa3ef831-3b27-4a0a-9f3d-15cf0a774b94 -i td/osd-scrub-repair/2/new.json 2026-02-22T20:52:39.888 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:52:39.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/2/new.json 2026-02-22T20:52:39.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXbJtpQrToKhAA8UjgAOKQmbUbVu0X2VVAAw== --osd-uuid fa3ef831-3b27-4a0a-9f3d-15cf0a774b94 2026-02-22T20:52:39.915 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:39.904+0000 7fa889d6da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:39.917 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:39.908+0000 7fa889d6da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:39.918 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:39.908+0000 7fa889d6da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:39.919 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:39.908+0000 7fa889d6da40 -1 bdev(0x55b76c02ac00 td/osd-scrub-repair/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:52:39.919 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:39.908+0000 7fa889d6da40 -1 bluestore(td/osd-scrub-repair/2) _read_fsid unparsable uuid 2026-02-22T20:52:40.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/2/keyring 2026-02-22T20:52:40.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:52:40.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:52:40.574 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:52:40.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:52:40.779 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:52:40.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:52:40.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:40.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:52:40.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:52:40.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:52:40.804 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:40.792+0000 7fc42ce44a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:40.805 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:40.796+0000 7fc42ce44a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:40.806 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:40.796+0000 7fc42ce44a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:40.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:52:40.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:40.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:52:40.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:40.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:40.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:40.938 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:52:40.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:40.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:40.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:52:41.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:41.179 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:41.168+0000 7fc42ce44a40 -1 Falling back to public interface 2026-02-22T20:52:41.857 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:41.844+0000 7fc42ce44a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:52:42.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:42.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:42.101 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:52:42.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:42.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:42.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:52:42.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:43.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:43.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:43.259 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:52:43.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:52:43.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:43.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:52:43.418 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2685471790,v1:127.0.0.1:6819/2685471790] [v2:127.0.0.1:6820/2685471790,v1:127.0.0.1:6821/2685471790] exists,up fa3ef831-3b27-4a0a-9f3d-15cf0a774b94 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:918: unfound_erasure_coded: for id in $(seq 0 3) 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:919: unfound_erasure_coded: run_osd td/osd-scrub-repair 3 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:43.419 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:43.420 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:43.420 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:43.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:52:43.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:52:43.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:52:43.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1098cd20-a433-4d1c-a636-ca87b1a9e978 2026-02-22T20:52:43.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 1098cd20-a433-4d1c-a636-ca87b1a9e978' 2026-02-22T20:52:43.424 INFO:tasks.workunit.client.0.trial150.stdout:add osd3 1098cd20-a433-4d1c-a636-ca87b1a9e978 2026-02-22T20:52:43.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:52:43.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCbbJtpCL6hGRAA9Pkq7khFkmKD36cWio6NmQ== 2026-02-22T20:52:43.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCbbJtpCL6hGRAA9Pkq7khFkmKD36cWio6NmQ=="}' 2026-02-22T20:52:43.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1098cd20-a433-4d1c-a636-ca87b1a9e978 -i td/osd-scrub-repair/3/new.json 2026-02-22T20:52:43.599 INFO:tasks.workunit.client.0.trial150.stdout:3 2026-02-22T20:52:43.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-repair/3/new.json 2026-02-22T20:52:43.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCbbJtpCL6hGRAA9Pkq7khFkmKD36cWio6NmQ== --osd-uuid 1098cd20-a433-4d1c-a636-ca87b1a9e978 2026-02-22T20:52:43.626 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:43.616+0000 7f5999e93a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:43.628 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:43.616+0000 7f5999e93a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:43.629 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:43.620+0000 7f5999e93a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:43.629 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:43.620+0000 7f5999e93a40 -1 bdev(0x55cea2e1cc00 td/osd-scrub-repair/3/block) open stat got: (1) Operation not permitted 2026-02-22T20:52:43.629 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:43.620+0000 7f5999e93a40 -1 bluestore(td/osd-scrub-repair/3) _read_fsid unparsable uuid 2026-02-22T20:52:44.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-repair/3/keyring 2026-02-22T20:52:44.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:52:44.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-22T20:52:44.254 INFO:tasks.workunit.client.0.trial150.stdout:adding osd3 key to auth repository 2026-02-22T20:52:44.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-repair/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:52:44.455 INFO:tasks.workunit.client.0.trial150.stdout:start osd.3 2026-02-22T20:52:44.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-22T20:52:44.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:44.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:52:44.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:52:44.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:52:44.479 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:44.468+0000 7f453989aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:44.481 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:44.472+0000 7f453989aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:44.482 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:44.472+0000 7f453989aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:44.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-22T20:52:44.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:44.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:52:44.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:44.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:44.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:44.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:44.612 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:52:44.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:44.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:52:44.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:44.867 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:44.856+0000 7f453989aa40 -1 Falling back to public interface 2026-02-22T20:52:45.512 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:52:45.500+0000 7f453989aa40 -1 osd.3 0 log_to_monitors true 2026-02-22T20:52:45.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:45.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:45.775 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:52:45.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:45.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:45.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:52:45.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:46.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:46.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:46.935 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:52:46.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:52:46.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:46.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:52:47.096 INFO:tasks.workunit.client.0.trial150.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/948951168,v1:127.0.0.1:6827/948951168] [v2:127.0.0.1:6828/948951168,v1:127.0.0.1:6829/948951168] exists,up 1098cd20-a433-4d1c-a636-ca87b1a9e978 2026-02-22T20:52:47.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:47.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:47.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:47.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:922: unfound_erasure_coded: create_ec_pool ecpool true k=2 m=2 2026-02-22T20:52:47.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2522: create_ec_pool: local pool_name=ecpool 2026-02-22T20:52:47.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2523: create_ec_pool: shift 2026-02-22T20:52:47.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2524: create_ec_pool: local allow_overwrites=true 2026-02-22T20:52:47.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2525: create_ec_pool: shift 2026-02-22T20:52:47.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2527: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=2 2026-02-22T20:52:47.339 INFO:tasks.workunit.client.0.trial150.stderr:load: isa 2026-02-22T20:52:47.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2529: create_ec_pool: create_pool ecpool 1 1 erasure myprofile 2026-02-22T20:52:47.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create ecpool 1 1 erasure myprofile 2026-02-22T20:52:47.642 INFO:tasks.workunit.client.0.trial150.stderr:pool 'ecpool' created 2026-02-22T20:52:47.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:52:48.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2531: create_ec_pool: '[' true = true ']' 2026-02-22T20:52:48.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2532: create_ec_pool: ceph osd pool set ecpool allow_ec_overwrites true 2026-02-22T20:52:48.838 INFO:tasks.workunit.client.0.trial150.stderr:set pool 1 allow_ec_overwrites to true 2026-02-22T20:52:48.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2535: create_ec_pool: wait_for_clean 2026-02-22T20:52:48.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:52:48.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:52:48.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:52:48.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:52:48.850 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:52:48.850 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:52:48.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:52:48.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:52:48.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:52:48.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:52:48.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:52:48.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:52:48.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:52:48.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:52:48.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:52:49.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:52:49.117 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:52:49.118 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:52:49.118 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:52:49.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:52:49.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:49.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:52:49.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-22T20:52:49.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-22T20:52:49.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484' 2026-02-22T20:52:49.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:49.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:52:49.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-22T20:52:49.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-22T20:52:49.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672964' 2026-02-22T20:52:49.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:49.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:52:49.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-22T20:52:49.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-22T20:52:49.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672964 2-60129542147' 2026-02-22T20:52:49.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:49.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:52:49.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=77309411331 2026-02-22T20:52:49.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 77309411331 2026-02-22T20:52:49.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836484 1-42949672964 2-60129542147 3-77309411331' 2026-02-22T20:52:49.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:49.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-22T20:52:49.409 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:49.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:52:49.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836484 2026-02-22T20:52:49.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:49.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836484 2026-02-22T20:52:49.413 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836484 2026-02-22T20:52:49.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-22T20:52:49.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:49.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:52:49.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:52:50.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:52:50.574 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:50.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836484 2026-02-22T20:52:50.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:52:51.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:52:51.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:51.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836484 2026-02-22T20:52:51.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:51.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-22T20:52:51.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:51.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:52:51.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672964 2026-02-22T20:52:51.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:51.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672964 2026-02-22T20:52:51.902 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672964 2026-02-22T20:52:51.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-22T20:52:51.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:52:52.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672964 -lt 42949672964 2026-02-22T20:52:52.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:52.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-22T20:52:52.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:52.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:52:52.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-22T20:52:52.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:52.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-22T20:52:52.066 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542147 2026-02-22T20:52:52.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-22T20:52:52.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:52:52.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542147 -lt 60129542147 2026-02-22T20:52:52.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:52.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-77309411331 2026-02-22T20:52:52.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:52.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:52:52.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-77309411331 2026-02-22T20:52:52.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:52.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=77309411331 2026-02-22T20:52:52.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 77309411331' 2026-02-22T20:52:52.230 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.3 seq 77309411331 2026-02-22T20:52:52.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:52:52.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 77309411331 -lt 77309411331 2026-02-22T20:52:52.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:52:52.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:52.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:52.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:52:52.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:52.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:52.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:52.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:52.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:52.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:52.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:52.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:52:52.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:52.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:52.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:52.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:52:52.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:52:52.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:52:52.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2536: create_ec_pool: return 0 2026-02-22T20:52:52.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:924: unfound_erasure_coded: add_something td/osd-scrub-repair ecpool 2026-02-22T20:52:52.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:71: add_something: local dir=td/osd-scrub-repair 2026-02-22T20:52:52.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:72: add_something: local poolname=ecpool 2026-02-22T20:52:52.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:73: add_something: local obj=SOMETHING 2026-02-22T20:52:52.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:74: add_something: local scrub=noscrub 2026-02-22T20:52:52.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:76: add_something: '[' noscrub = noscrub ']' 2026-02-22T20:52:52.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:78: add_something: ceph osd set noscrub 2026-02-22T20:52:53.136 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:52:53.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:79: add_something: ceph osd set nodeep-scrub 2026-02-22T20:52:53.341 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:52:53.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:85: add_something: local payload=ABCDEF 2026-02-22T20:52:53.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:86: add_something: echo ABCDEF 2026-02-22T20:52:53.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:87: add_something: rados --pool ecpool put SOMETHING td/osd-scrub-repair/ORIGINAL 2026-02-22T20:52:53.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: unfound_erasure_coded: get_primary ecpool SOMETHING 2026-02-22T20:52:53.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=ecpool 2026-02-22T20:52:53.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T20:52:53.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:52:53.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:52:53.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:926: unfound_erasure_coded: local primary=1 2026-02-22T20:52:53.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: get_osds ecpool SOMETHING 2026-02-22T20:52:53.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: sed -e s/1// 2026-02-22T20:52:53.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=ecpool 2026-02-22T20:52:53.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=SOMETHING 2026-02-22T20:52:53.540 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:52:53.541 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-22T20:52:53.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-22T20:52:53.706 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T20:52:53.706 INFO:tasks.workunit.client.0.trial150.stderr:2 2026-02-22T20:52:53.706 INFO:tasks.workunit.client.0.trial150.stderr:3' 2026-02-22T20:52:53.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 3 2026-02-22T20:52:53.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: osds=('0' '2' '3') 2026-02-22T20:52:53.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:927: unfound_erasure_coded: local -a osds 2026-02-22T20:52:53.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:928: unfound_erasure_coded: local not_primary_first=0 2026-02-22T20:52:53.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:929: unfound_erasure_coded: local not_primary_second=2 2026-02-22T20:52:53.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:930: unfound_erasure_coded: local not_primary_third=3 2026-02-22T20:52:53.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:935: unfound_erasure_coded: pids= 2026-02-22T20:52:53.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:936: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:52:53.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:52:53.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:52:53.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 456360"' 2026-02-22T20:52:53.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 456360' 2026-02-22T20:52:53.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:937: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:52:53.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:52:53.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:52:53.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 456362"' 2026-02-22T20:52:53.711 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 456362' 2026-02-22T20:52:53.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:52:53.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:938: unfound_erasure_coded: run_in_background pids objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:52:53.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: run_in_background: local pid_variable=pids 2026-02-22T20:52:53.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: run_in_background: shift 2026-02-22T20:52:53.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/456363: /' 2026-02-22T20:52:53.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: eval 'pids+=" 456365"' 2026-02-22T20:52:53.711 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: run_in_background: pids+=' 456365' 2026-02-22T20:52:53.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/456366: /' 2026-02-22T20:52:53.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:52:53.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:939: unfound_erasure_coded: wait_background pids 2026-02-22T20:52:53.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: wait_background: pids=' 456360 456362 456365' 2026-02-22T20:52:53.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: wait_background: return_code=0 2026-02-22T20:52:53.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:52:53.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 456360 2026-02-22T20:52:53.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: objectstore_tool td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:52:53.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: sed 's/^/456369: /' 2026-02-22T20:52:54.341 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:52:54.341 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:52:54.341 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-22T20:52:54.341 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:52:54.341 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:52:54.342 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:52:54.342 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:52:54.342 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-22T20:52:54.342 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:52:54.342 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.0 2026-02-22T20:52:54.342 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:52:54.343 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:52:54.343 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:52:54.343 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:52:54.343 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:52:54.343 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:52:54.343 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 0 SOMETHING remove 2026-02-22T20:52:54.343 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:52:54.344 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:52:54.344 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-22T20:52:54.344 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:52:54.344 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:52:54.344 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/0 SOMETHING remove 2026-02-22T20:52:54.344 INFO:tasks.workunit.client.0.trial150.stderr:456363: remove 1#1:eb822e21:::SOMETHING:head# 2026-02-22T20:52:54.344 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 0 2026-02-22T20:52:54.344 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:54.344 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:52:54.344 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:52:54.345 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:52:54.345 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/0 2026-02-22T20:52:54.345 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:54.345 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:54.346 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:54.346 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:54.346 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/0' 2026-02-22T20:52:54.346 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/0/journal' 2026-02-22T20:52:54.346 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:54.347 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:52:54.347 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:54.347 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:52:54.347 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:54.347 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:54.347 INFO:tasks.workunit.client.0.trial150.stderr:456363: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:54.347 INFO:tasks.workunit.client.0.trial150.stderr:456363: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:54.347 INFO:tasks.workunit.client.0.trial150.stderr:456363: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:54.347 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:54.347 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:54.348 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:54.348 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:54.348 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:54.348 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:54.348 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:54.348 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:54.348 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:54.348 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:52:54.348 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:52:54.349 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/0 2026-02-22T20:52:54.349 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:52:54.349 INFO:tasks.workunit.client.0.trial150.stderr:456363: start osd.0 2026-02-22T20:52:54.349 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/0 --osd-journal=td/osd-scrub-repair/0/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:54.349 INFO:tasks.workunit.client.0.trial150.stderr:4456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:52:54.350 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:52:54.350 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=3 2026-02-22T20:52:54.350 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:52:54.350 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:52:54.350 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:52:54.350 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:52:54.350 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=3 2026-02-22T20:52:54.350 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:52:54.351 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.3 2026-02-22T20:52:54.351 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:52:54.351 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:52:54.351 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:52:54.351 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:52:54.351 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:52:54.351 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:52:54.351 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 3 SOMETHING remove 2026-02-22T20:52:54.351 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:52:54.351 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:52:54.352 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=3 2026-02-22T20:52:54.352 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:52:54.352 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:52:54.352 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/3 SOMETHING remove 2026-02-22T20:52:54.352 INFO:tasks.workunit.client.0.trial150.stderr:456369: remove 3#1:eb822e21:::SOMETHING:head# 2026-02-22T20:52:54.352 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 3 2026-02-22T20:52:54.352 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:54.352 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:52:54.352 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2026-02-22T20:52:54.352 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:52:54.353 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/3 2026-02-22T20:52:54.353 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:54.353 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:54.353 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:54.353 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:54.354 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/3' 2026-02-22T20:52:54.354 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/3/journal' 2026-02-22T20:52:54.354 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:54.354 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:52:54.354 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:54.354 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:52:54.354 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:54.354 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:54.354 INFO:tasks.workunit.client.0.trial150.stderr:456369: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:54.355 INFO:tasks.workunit.client.0.trial150.stderr:456369: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:54.355 INFO:tasks.workunit.client.0.trial150.stderr:456369: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:54.355 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:54.355 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:54.355 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:54.355 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:54.355 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:54.355 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:54.355 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:54.355 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:54.356 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:54.356 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:52:54.356 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:52:54.356 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/3 2026-02-22T20:52:54.356 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2026-02-22T20:52:54.356 INFO:tasks.workunit.client.0.trial150.stderr:456369: start osd.3 2026-02-22T20:52:54.356 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/3 --osd-journal=td/osd-scrub-repair/3/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:54.417 INFO:tasks.workunit.client.0.trial150.stderr:4456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-repair 2026-02-22T20:52:54.417 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T20:52:54.417 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T20:52:54.418 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T20:52:54.418 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:52:54.418 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-repair 2026-02-22T20:52:54.418 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T20:52:54.418 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T20:52:54.418 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T20:52:54.418 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-repair TERM osd.2 2026-02-22T20:52:54.418 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:52:54.418 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:52:54.418 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:52:54.419 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:52:54.419 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:52:54.419 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:52:54.419 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-repair 2 SOMETHING remove 2026-02-22T20:52:54.419 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-repair 2026-02-22T20:52:54.419 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T20:52:54.419 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T20:52:54.419 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T20:52:54.419 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:52:54.419 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-repair/2 SOMETHING remove 2026-02-22T20:52:54.419 INFO:tasks.workunit.client.0.trial150.stderr:456366: remove 2#1:eb822e21:::SOMETHING:head# 2026-02-22T20:52:54.419 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-repair 2 2026-02-22T20:52:54.420 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-repair 2026-02-22T20:52:54.420 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:52:54.420 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T20:52:54.420 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:52:54.420 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-repair/2 2026-02-22T20:52:54.420 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false ' 2026-02-22T20:52:54.420 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:52:54.421 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:52:54.421 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:52:54.421 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-repair/2' 2026-02-22T20:52:54.421 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-repair/2/journal' 2026-02-22T20:52:54.422 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:52:54.422 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:52:54.422 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-repair' 2026-02-22T20:52:54.422 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:52:54.422 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:52:54.422 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:52:54.422 INFO:tasks.workunit.client.0.trial150.stderr:456366: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:52:54.422 INFO:tasks.workunit.client.0.trial150.stderr:456366: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:52:54.423 INFO:tasks.workunit.client.0.trial150.stderr:456366: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:52:54.423 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:54.423 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' 2026-02-22T20:52:54.423 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:52:54.423 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-repair/$name.log' 2026-02-22T20:52:54.423 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-repair/$name.pid' 2026-02-22T20:52:54.423 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:52:54.423 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:52:54.423 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:52:54.423 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:52:54.424 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:52:54.424 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:52:54.424 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-repair/2 2026-02-22T20:52:54.424 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T20:52:54.424 INFO:tasks.workunit.client.0.trial150.stderr:456366: start osd.2 2026-02-22T20:52:54.424 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=470fd38f-b5b4-490c-bcf4-4e2f1393e8de --auth-supported=none --mon-host=127.0.0.1:7107 --osd-skip-data-digest=false --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-repair/2 --osd-journal=td/osd-scrub-repair/2/journal --chdir= --run-dir=td/osd-scrub-repair '--admin-socket=/tmp/ceph-asok.40073/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-repair/$name.log' '--pid-file=td/osd-scrub-repair/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:52:55.832 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/0/whoami 2026-02-22T20:52:55.833 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:52:55.833 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:52:55.833 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:52:55.833 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:52:55.833 INFO:tasks.workunit.client.0.trial150.stderr:456363: 2026-02-22T20:52:54.352+0000 7f14afae7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:55.833 INFO:tasks.workunit.client.0.trial150.stderr:456363: 2026-02-22T20:52:54.356+0000 7f14afae7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:55.833 INFO:tasks.workunit.client.0.trial150.stderr:456363: 2026-02-22T20:52:54.356+0000 7f14afae7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:55.833 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:52:55.833 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:55.833 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:52:55.834 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:55.834 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:55.834 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:55.834 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:55.834 INFO:tasks.workunit.client.0.trial150.stderr:456363: 0 2026-02-22T20:52:55.834 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:55.834 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:52:55.834 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:55.834 INFO:tasks.workunit.client.0.trial150.stderr:456363: 2026-02-22T20:52:54.824+0000 7f14afae7a40 -1 Falling back to public interface 2026-02-22T20:52:55.834 INFO:tasks.workunit.client.0.trial150.stderr:456363: 2026-02-22T20:52:55.592+0000 7f14afae7a40 -1 osd.0 26 log_to_monitors true 2026-02-22T20:52:55.834 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:55.834 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:55.835 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:55.835 INFO:tasks.workunit.client.0.trial150.stderr:456363: 1 2026-02-22T20:52:55.835 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:55.835 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:52:55.835 INFO:tasks.workunit.client.0.trial150.stderr:456363: osd.0 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [5,26) [v2:127.0.0.1:6802/1088323983,v1:127.0.0.1:6803/1088323983] [v2:127.0.0.1:6804/1088323983,v1:127.0.0.1:6805/1088323983] exists,up a27e3098-92ce-4822-8e46-b46bec819d7d 2026-02-22T20:52:55.835 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:55.835 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:55.835 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:55.835 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:52:55.835 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:52:55.835 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:52:55.835 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:52:55.836 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:52:55.836 INFO:tasks.workunit.client.0.trial150.stderr:456363: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:52:55.842 INFO:tasks.workunit.client.0.trial150.stderr:456363: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get56369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/3/whoami 2026-02-22T20:52:55.842 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2026-02-22T20:52:55.842 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:52:55.843 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:52:55.843 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:52:55.843 INFO:tasks.workunit.client.0.trial150.stderr:456369: 2026-02-22T20:52:54.356+0000 7f5f65b45a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:55.843 INFO:tasks.workunit.client.0.trial150.stderr:456369: 2026-02-22T20:52:54.356+0000 7f5f65b45a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:55.843 INFO:tasks.workunit.client.0.trial150.stderr:456369: 2026-02-22T20:52:54.360+0000 7f5f65b45a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:55.843 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2026-02-22T20:52:55.843 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:55.843 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-22T20:52:55.843 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:55.843 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:55.843 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:55.843 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: 0 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: 2026-02-22T20:52:54.836+0000 7f5f65b45a40 -1 Falling back to public interface 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: 2026-02-22T20:52:55.572+0000 7f5f65b45a40 -1 osd.3 26 log_to_monitors true 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: 1 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-22T20:52:55.844 INFO:tasks.workunit.client.0.trial150.stderr:456369: osd.3 up in weight 1 up_from 30 up_thru 0 down_at 27 last_clean_interval [18,26) [v2:127.0.0.1:6818/965558391,v1:127.0.0.1:6819/965558391] [v2:127.0.0.1:6820/965558391,v1:127.0.0.1:6821/965558391] exists,up 1098cd20-a433-4d1c-a636-ca87b1a9e978 2026-02-22T20:52:55.845 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:55.845 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:55.845 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:55.845 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:52:55.845 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:52:55.845 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:52:55.845 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:52:55.845 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:52:55.845 INFO:tasks.workunit.client.0.trial150.stderr:456369: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:52:55.908 INFO:tasks.workunit.client.0.trial150.stderr:456369: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_ti56366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-repair/2/whoami 2026-02-22T20:52:55.909 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T20:52:55.909 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:52:55.909 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:52:55.909 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:52:55.909 INFO:tasks.workunit.client.0.trial150.stderr:456366: 2026-02-22T20:52:54.424+0000 7f80f13c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:55.909 INFO:tasks.workunit.client.0.trial150.stderr:456366: 2026-02-22T20:52:54.424+0000 7f80f13c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:55.909 INFO:tasks.workunit.client.0.trial150.stderr:456366: 2026-02-22T20:52:54.428+0000 7f80f13c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:52:55.909 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T20:52:55.909 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:52:55.909 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:52:55.910 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:52:55.910 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:52:55.910 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:55.910 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:52:55.910 INFO:tasks.workunit.client.0.trial150.stderr:456366: 0 2026-02-22T20:52:55.910 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:55.910 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:52:55.910 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:52:55.910 INFO:tasks.workunit.client.0.trial150.stderr:456366: 2026-02-22T20:52:54.888+0000 7f80f13c4a40 -1 Falling back to public interface 2026-02-22T20:52:55.910 INFO:tasks.workunit.client.0.trial150.stderr:456366: 2026-02-22T20:52:55.640+0000 7f80f13c4a40 -1 osd.2 26 log_to_monitors true 2026-02-22T20:52:55.910 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:52:55.910 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: 1 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: osd.2 up in weight 1 up_from 31 up_thru 0 down_at 27 last_clean_interval [14,26) [v2:127.0.0.1:6826/303036812,v1:127.0.0.1:6827/303036812] [v2:127.0.0.1:6828/303036812,v1:127.0.0.1:6829/303036812] exists,up fa3ef831-3b27-4a0a-9f3d-15cf0a774b94 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:52:55.911 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:52:55.912 INFO:tasks.workunit.client.0.trial150.stderr:456366: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456366: ///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_ti_timeout_delays: echo true 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: 1 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: 2 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: 3' 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882' 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672966 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672966 2026-02-22T20:52:56.405 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-42949672966' 2026-02-22T20:52:56.406 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:56.406 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:52:56.406 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986178 2026-02-22T20:52:56.406 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986178 2026-02-22T20:52:56.406 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-42949672966 2-133143986178' 2026-02-22T20:52:56.406 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:56.406 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:52:56.406 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018882 2026-02-22T20:52:56.406 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018882 2026-02-22T20:52:56.406 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018882 1-42949672966 2-133143986178 3-128849018882' 2026-02-22T20:52:56.406 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:56.406 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018882 2026-02-22T20:52:56.420 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/meout_delays: echo true 2026-02-22T20:52:56.420 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:52:56.420 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:52:56.420 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:52:56.420 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: 1 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: 2 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: 3' 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018883 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018883 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018883' 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018883 1-42949672967' 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986179 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986179 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018883 1-42949672967 2-133143986179' 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018883 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018883 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018883 1-42949672967 2-133143986179 3-128849018883' 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:56.421 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018883 2026-02-22T20:52:56.492 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubumeout_delays: echo true 2026-02-22T20:52:56.492 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:52:56.492 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:52:56.492 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:52:56.492 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:52:56.492 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:52:56.492 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:52:56.492 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:52:56.492 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: 1 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: 2 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: 3' 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018884 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018884 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884' 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672968 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672968 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884 1-42949672968' 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=133143986180 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 133143986180 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884 1-42949672968 2-133143986180' 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=128849018884 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 128849018884 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-128849018884 1-42949672968 2-133143986180 3-128849018884' 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:56.493 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-128849018884 2026-02-22T20:52:58.086 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:58.086 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:52:58.086 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018883 2026-02-22T20:52:58.086 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:58.086 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018883 2026-02-22T20:52:58.087 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018883' 2026-02-22T20:52:58.087 INFO:tasks.workunit.client.0.trial150.stderr:456369: waiting osd.0 seq 128849018883 2026-02-22T20:52:58.087 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:58.087 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 128849018883 2026-02-22T20:52:58.087 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:52:58.087 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:52:58.087 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:58.087 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018883 2026-02-22T20:52:58.087 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:58.087 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-22T20:52:58.087 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:58.087 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:52:58.088 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-22T20:52:58.088 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:58.088 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-22T20:52:58.088 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-22T20:52:58.088 INFO:tasks.workunit.client.0.trial150.stderr:456369: waiting osd.1 seq 42949672967 2026-02-22T20:52:58.088 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:52:58.088 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672967 2026-02-22T20:52:58.088 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:58.088 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-133143986179 2026-02-22T20:52:58.088 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:58.088 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:52:58.088 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-133143986179 2026-02-22T20:52:58.089 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:58.089 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986179 2026-02-22T20:52:58.089 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 133143986179' 2026-02-22T20:52:58.089 INFO:tasks.workunit.client.0.trial150.stderr:456369: waiting osd.2 seq 133143986179 2026-02-22T20:52:58.089 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:52:58.089 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986179 2026-02-22T20:52:58.089 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:58.091 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:22ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:58.091 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:52:58.091 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018882 2026-02-22T20:52:58.091 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:58.091 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-22T20:52:58.091 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018882' 2026-02-22T20:52:58.091 INFO:tasks.workunit.client.0.trial150.stderr:456363: waiting osd.0 seq 128849018882 2026-02-22T20:52:58.091 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:58.091 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 128849018882 2026-02-22T20:52:58.091 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:52:58.091 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018882 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672966 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672966 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672966 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: waiting osd.1 seq 42949672966 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:52:58.092 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672966 2026-02-22T20:52:58.093 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:58.093 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-133143986178 2026-02-22T20:52:58.093 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:58.093 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:52:58.093 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-133143986178 2026-02-22T20:52:58.093 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:58.093 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986178 2026-02-22T20:52:58.093 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 133143986178' 2026-02-22T20:52:58.093 INFO:tasks.workunit.client.0.trial150.stderr:456363: waiting osd.2 seq 133143986178 2026-02-22T20:52:58.093 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:52:58.093 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986178 2026-02-22T20:52:58.093 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:58.169 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.shntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:58.169 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:52:58.169 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-128849018884 2026-02-22T20:52:58.169 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:58.169 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018884 2026-02-22T20:52:58.170 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 128849018884' 2026-02-22T20:52:58.170 INFO:tasks.workunit.client.0.trial150.stderr:456366: waiting osd.0 seq 128849018884 2026-02-22T20:52:58.170 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:58.170 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 128849018884 2026-02-22T20:52:58.170 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:52:58.170 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:52:58.170 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:52:58.170 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018884 2026-02-22T20:52:58.170 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:58.170 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672968 2026-02-22T20:52:58.170 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:58.170 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672968 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672968 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: waiting osd.1 seq 42949672968 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672968 -lt 42949672968 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-133143986180 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-133143986180 2026-02-22T20:52:58.171 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:58.172 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=133143986180 2026-02-22T20:52:58.172 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 133143986180' 2026-02-22T20:52:58.172 INFO:tasks.workunit.client.0.trial150.stderr:456366: waiting osd.2 seq 133143986180 2026-02-22T20:52:58.172 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:52:58.172 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 133143986180 -lt 133143986180 2026-02-22T20:52:58.172 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:52:58.808 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-128849018883 2026-02-22T20:52:58.808 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:58.809 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:52:58.809 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-128849018883 2026-02-22T20:52:58.809 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:58.809 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018883 2026-02-22T20:52:58.809 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 128849018883' 2026-02-22T20:52:58.809 INFO:tasks.workunit.client.0.trial150.stderr:456369: waiting osd.3 seq 128849018883 2026-02-22T20:52:58.809 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:52:58.809 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018883 2026-02-22T20:52:58.809 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:52:58.809 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:58.809 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:58.810 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:52:58.810 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:58.810 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:58.810 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:58.810 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:58.810 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:58.810 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:58.810 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:58.810 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:52:58.810 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:58.810 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:58.810 INFO:tasks.workunit.client.0.trial150.stderr:456369: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:58.811 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:52:58.811 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:52:58.811 INFO:tasks.workunit.client.0.trial150.stderr:456369: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:52:58.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:52:58.813 INFO:tasks.workunit.client.0.trial150.stderr::2267: flush_pg_stats: echo 3-128849018882 2026-02-22T20:52:58.814 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:58.814 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:52:58.814 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-128849018882 2026-02-22T20:52:58.814 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:58.814 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018882 2026-02-22T20:52:58.814 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 128849018882' 2026-02-22T20:52:58.814 INFO:tasks.workunit.client.0.trial150.stderr:456363: waiting osd.3 seq 128849018882 2026-02-22T20:52:58.814 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:52:58.814 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018882 2026-02-22T20:52:58.814 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:52:58.814 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:58.815 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:52:58.816 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:52:58.816 INFO:tasks.workunit.client.0.trial150.stderr:456363: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:52:58.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:52:58.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:52:58.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 456362 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:67: flush_pg_stats: echo 3-128849018884 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=3 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 3-128849018884 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=128849018884 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.3 seq 128849018884' 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: waiting osd.3 seq 128849018884 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 128849018884 -lt 128849018884 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: //home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:52:58.900 INFO:tasks.workunit.client.0.trial150.stderr:456366: /home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: run_in_background: return 0 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: wait_background: for pid in $pids 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: wait_background: wait 456365 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: eval 'pids='\'''\''' 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: wait_background: pids= 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: wait_background: return 0 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:940: unfound_erasure_coded: return_code=0 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:941: unfound_erasure_coded: '[' 0 -ne 0 ']' 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: unfound_erasure_coded: get_pg ecpool SOMETHING 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=ecpool 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map ecpool SOMETHING 2026-02-22T20:52:58.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T20:52:59.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:946: unfound_erasure_coded: local pg=1.0 2026-02-22T20:52:59.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:947: unfound_erasure_coded: repair 1.0 2026-02-22T20:52:59.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-22T20:52:59.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-22T20:52:59.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:52:59.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:52:59.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:52:59.065 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:52:59.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T20:52:47.636501+0000 2026-02-22T20:52:59.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-22T20:52:59.374 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0s0 on osd.1 to repair 2026-02-22T20:52:59.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-22T20:52:47.636501+0000 2026-02-22T20:52:59.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:52:59.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:52:47.636501+0000 2026-02-22T20:52:59.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:52:59.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:52:59.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:52:59.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:52:59.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:52:59.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:52:59.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:52:59.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:52:59.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:52:47.636501+0000 '>' 2026-02-22T20:52:47.636501+0000 2026-02-22T20:52:59.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:53:00.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:53:00.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:53:00.546 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:53:00.546 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:53:00.546 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:53:00.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:53:00.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:53:00.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:52:47.636501+0000 '>' 2026-02-22T20:52:47.636501+0000 2026-02-22T20:53:00.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:53:01.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:53:01.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:53:01.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:53:01.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:53:01.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:53:01.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:53:01.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:53:01.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:52:47.636501+0000 '>' 2026-02-22T20:52:47.636501+0000 2026-02-22T20:53:01.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:53:02.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:53:02.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:53:02.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:53:02.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:53:02.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:53:02.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:53:02.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:53:03.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:52:47.636501+0000 '>' 2026-02-22T20:52:47.636501+0000 2026-02-22T20:53:03.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:53:04.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:53:04.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:53:04.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:53:04.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:53:04.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:53:04.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:53:04.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:53:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:52:47.636501+0000 '>' 2026-02-22T20:52:47.636501+0000 2026-02-22T20:53:04.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:53:05.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:53:05.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:53:05.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:53:05.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:53:05.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:53:05.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:53:05.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:53:05.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:53:00.163083+0000 '>' 2026-02-22T20:52:47.636501+0000 2026-02-22T20:53:05.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:53:05.348 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: unfound_erasure_coded: seq 1 60 2026-02-22T20:53:05.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:952: unfound_erasure_coded: for f in `seq 1 60` 2026-02-22T20:53:05.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: ceph -s 2026-02-22T20:53:05.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: grep '1/1 objects unfound' 2026-02-22T20:53:05.538 INFO:tasks.workunit.client.0.trial150.stdout: 1/1 objects unfound (100.000%) 2026-02-22T20:53:05.538 INFO:tasks.workunit.client.0.trial150.stdout: 1/1 objects unfound (100.000%) 2026-02-22T20:53:05.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:953: unfound_erasure_coded: break 2026-02-22T20:53:05.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: unfound_erasure_coded: ceph -s 2026-02-22T20:53:05.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:956: unfound_erasure_coded: grep '4 up' 2026-02-22T20:53:05.731 INFO:tasks.workunit.client.0.trial150.stdout: osd: 4 osds: 4 up (since 9s), 4 in (since 22s) 2026-02-22T20:53:05.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: unfound_erasure_coded: ceph -s 2026-02-22T20:53:05.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:957: unfound_erasure_coded: grep '4 in' 2026-02-22T20:53:05.919 INFO:tasks.workunit.client.0.trial150.stdout: osd: 4 osds: 4 up (since 10s), 4 in (since 22s) 2026-02-22T20:53:05.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: unfound_erasure_coded: ceph -s 2026-02-22T20:53:05.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:958: unfound_erasure_coded: grep '1/1 objects unfound' 2026-02-22T20:53:06.108 INFO:tasks.workunit.client.0.trial150.stdout: 1/1 objects unfound (100.000%) 2026-02-22T20:53:06.108 INFO:tasks.workunit.client.0.trial150.stdout: 1/1 objects unfound (100.000%) 2026-02-22T20:53:06.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-repair.sh:66: run: teardown td/osd-scrub-repair 2026-02-22T20:53:06.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:53:06.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:53:06.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:53:06.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:53:06.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:53:06.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:53:06.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:53:06.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:53:06.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:53:06.223 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:53:06.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:53:06.224 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:53:06.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:53:06.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:53:06.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:53:06.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:53:06.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:53:06.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:53:06.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:53:06.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:53:06.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:53:06.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:53:06.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:53:06.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:06.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:53:06.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:53:06.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:53:06.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:53:06.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-22T20:53:06.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-scrub-repair 0 2026-02-22T20:53:06.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-repair 2026-02-22T20:53:06.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-22T20:53:06.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-repair KILL 2026-02-22T20:53:06.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:53:06.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:53:06.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:53:06.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:53:06.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:53:06.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:53:06.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:53:06.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:53:06.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:53:06.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:53:06.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:53:06.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:53:06.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:53:06.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:53:06.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:53:06.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:53:06.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:53:06.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T20:53:06.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-repair 2026-02-22T20:53:06.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:53:06.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:06.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.40073 2026-02-22T20:53:06.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.40073 2026-02-22T20:53:06.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:53:06.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:53:06.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-22T20:53:06.262 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T20:53:06.262 DEBUG:teuthology.orchestra.run.trial150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T20:53:06.312 INFO:tasks.workunit:Running workunit scrub/osd-scrub-snaps.sh... 2026-02-22T20:53:06.312 DEBUG:teuthology.orchestra.run.trial150:workunit test scrub/osd-scrub-snaps.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5c3084f688b1f6eff223b8d259a8316f9844c8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh 2026-02-22T20:53:06.361 INFO:tasks.workunit.client.0.trial150.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T20:53:06.367 INFO:tasks.workunit.client.0.trial150.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T20:53:06.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T20:53:06.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T20:53:06.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-22T20:53:06.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-22T20:53:06.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-22T20:53:06.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-22T20:53:06.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-22T20:53:06.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-22T20:53:06.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-scrub-snaps 2026-02-22T20:53:06.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:27: run: local dir=td/osd-scrub-snaps 2026-02-22T20:53:06.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:28: run: shift 2026-02-22T20:53:06.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: export CEPH_MON=127.0.0.1:7121 2026-02-22T20:53:06.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:30: run: CEPH_MON=127.0.0.1:7121 2026-02-22T20:53:06.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:31: run: export CEPH_ARGS 2026-02-22T20:53:06.368 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: uuidgen 2026-02-22T20:53:06.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:32: run: CEPH_ARGS+='--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none ' 2026-02-22T20:53:06.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:33: run: CEPH_ARGS+='--mon-host=127.0.0.1:7121 ' 2026-02-22T20:53:06.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:35: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-22T20:53:06.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: set 2026-02-22T20:53:06.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:36: run: local 'funcs=TEST_scrub_snaps 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:TEST_scrub_snaps_primary 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:TEST_scrub_snaps_replica' 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:53:06.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:53:06.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:53:06.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:53:06.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:53:06.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:53:06.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:53:06.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:53:06.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:53:06.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:53:06.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:53:06.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:53:06.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:53:06.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:53:06.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:53:06.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-22T20:53:06.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:53:06.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:06.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:06.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.460661 2026-02-22T20:53:06.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:53:06.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:53:06.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2026-02-22T20:53:06.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:53:06.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:06.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:06.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.460661 2026-02-22T20:53:06.387 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:53:06.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-22T20:53:06.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-22T20:53:06.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:53:06.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2026-02-22T20:53:06.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps td/osd-scrub-snaps 2026-02-22T20:53:06.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:171: TEST_scrub_snaps: local dir=td/osd-scrub-snaps 2026-02-22T20:53:06.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:172: TEST_scrub_snaps: local poolname=test 2026-02-22T20:53:06.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:173: TEST_scrub_snaps: local OBJS=16 2026-02-22T20:53:06.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:174: TEST_scrub_snaps: local OSDS=1 2026-02-22T20:53:06.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:176: TEST_scrub_snaps: TESTDATA=testdata.460661 2026-02-22T20:53:06.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:178: TEST_scrub_snaps: run_mon td/osd-scrub-snaps a --osd_pool_default_size=1 2026-02-22T20:53:06.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2026-02-22T20:53:06.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:53:06.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:53:06.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:53:06.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2026-02-22T20:53:06.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=1 2026-02-22T20:53:06.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:53:06.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:53:06.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:53:06.411 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:53:06.411 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:06.411 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:06.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:53:06.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 2026-02-22T20:53:06.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:53:06.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:53:06.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:53:06.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:53:06.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:53:06.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:53:06.443 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:53:06.443 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:53:06.443 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:53:06.443 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:53:06.443 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:06.443 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:06.443 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.460661/ceph-mon.a.asok 2026-02-22T20:53:06.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:53:06.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.460661/ceph-mon.a.asok config get fsid 2026-02-22T20:53:06.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:53:06.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:53:06.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:53:06.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:53:06.503 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:53:06.503 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:53:06.503 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:53:06.503 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:53:06.503 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:53:06.503 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:06.503 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:06.504 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.460661/ceph-mon.a.asok 2026-02-22T20:53:06.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:53:06.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.460661/ceph-mon.a.asok config get mon_host 2026-02-22T20:53:06.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:179: TEST_scrub_snaps: run_mgr td/osd-scrub-snaps x 2026-02-22T20:53:06.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2026-02-22T20:53:06.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:53:06.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:53:06.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:53:06.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2026-02-22T20:53:06.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:53:06.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:53:06.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:53:06.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:53:06.656 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:53:06.656 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:06.656 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:06.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:53:06.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:53:06.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:53:06.673 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: expr 1 - 1 2026-02-22T20:53:06.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: seq 0 0 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:180: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:182: TEST_scrub_snaps: run_osd td/osd-scrub-snaps 0 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:53:06.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:53:06.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-22T20:53:06.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:53:06.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a7594624-9f48-4972-bc0c-05d4b36c7e0b 2026-02-22T20:53:06.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a7594624-9f48-4972-bc0c-05d4b36c7e0b' 2026-02-22T20:53:06.678 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 a7594624-9f48-4972-bc0c-05d4b36c7e0b 2026-02-22T20:53:06.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:53:06.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCybJtpu97BKBAAxO7ySjagWiNF0/ZUuGDmMw== 2026-02-22T20:53:06.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCybJtpu97BKBAAxO7ySjagWiNF0/ZUuGDmMw=="}' 2026-02-22T20:53:06.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a7594624-9f48-4972-bc0c-05d4b36c7e0b -i td/osd-scrub-snaps/0/new.json 2026-02-22T20:53:06.780 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:53:06.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2026-02-22T20:53:06.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCybJtpu97BKBAAxO7ySjagWiNF0/ZUuGDmMw== --osd-uuid a7594624-9f48-4972-bc0c-05d4b36c7e0b 2026-02-22T20:53:06.803 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:06.792+0000 7f1d82d36a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:06.805 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:06.796+0000 7f1d82d36a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:06.806 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:06.796+0000 7f1d82d36a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:06.806 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:06.796+0000 7f1d82d36a40 -1 bdev(0x5557afc06c00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:53:06.806 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:06.796+0000 7f1d82d36a40 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2026-02-22T20:53:07.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2026-02-22T20:53:07.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:53:07.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:53:07.572 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:53:07.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:53:07.768 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:53:07.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:53:07.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:53:07.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:53:07.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:53:07.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:53:07.791 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:07.780+0000 7f20359c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:07.792 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:07.780+0000 7f20359c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:07.793 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:07.784+0000 7f20359c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:07.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:53:07.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:53:07.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:53:07.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:53:07.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:53:07.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:53:07.927 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:53:07.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:53:07.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:53:07.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:53:08.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:53:08.198 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:08.188+0000 7f20359c4a40 -1 Falling back to public interface 2026-02-22T20:53:08.739 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:08.728+0000 7f20359c4a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:53:09.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:53:09.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:53:09.092 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:53:09.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:53:09.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:53:09.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:53:09.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:53:10.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:53:10.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:53:10.253 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:53:10.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:53:10.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:53:10.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:53:10.410 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/667257433,v1:127.0.0.1:6803/667257433] [v2:127.0.0.1:6804/667257433,v1:127.0.0.1:6805/667257433] exists,up a7594624-9f48-4972-bc0c-05d4b36c7e0b 2026-02-22T20:53:10.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:53:10.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:53:10.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:53:10.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:186: TEST_scrub_snaps: ceph osd set noscrub 2026-02-22T20:53:10.611 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:53:10.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:187: TEST_scrub_snaps: ceph osd set nodeep-scrub 2026-02-22T20:53:10.814 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:53:10.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:190: TEST_scrub_snaps: create_pool test 1 1 2026-02-22T20:53:10.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-22T20:53:11.017 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T20:53:11.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:53:12.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:191: TEST_scrub_snaps: wait_for_clean 2026-02-22T20:53:12.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:53:12.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:53:12.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:53:12.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:53:12.031 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:53:12.031 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:53:12.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:53:12.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:53:12.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:53:12.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:53:12.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:53:12.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:53:12.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:53:12.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:53:12.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:53:12.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:53:12.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:53:12.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:53:12.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:53:12.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836482 2026-02-22T20:53:12.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836482 2026-02-22T20:53:12.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836482' 2026-02-22T20:53:12.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:53:12.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836482 2026-02-22T20:53:12.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:53:12.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:53:12.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836482 2026-02-22T20:53:12.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:53:12.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836482 2026-02-22T20:53:12.359 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836482 2026-02-22T20:53:12.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836482' 2026-02-22T20:53:12.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:53:12.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836481 -lt 21474836482 2026-02-22T20:53:12.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:53:13.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:53:13.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:53:13.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836482 2026-02-22T20:53:13.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:53:13.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:53:13.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:53:13.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:53:13.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:53:13.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:53:13.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:53:13.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:53:13.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:53:13.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:53:13.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:53:14.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:53:14.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:53:14.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:53:14.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:53:14.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:53:14.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:53:14.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:53:14.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: ceph osd dump 2026-02-22T20:53:14.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: grep '^pool.*['\'']test['\'']' 2026-02-22T20:53:14.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: awk '{ print $2 }' 2026-02-22T20:53:14.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:192: TEST_scrub_snaps: poolid=1 2026-02-22T20:53:14.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:194: TEST_scrub_snaps: dd if=/dev/urandom of=testdata.460661 bs=1032 count=1 2026-02-22T20:53:14.394 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T20:53:14.394 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T20:53:14.394 INFO:tasks.workunit.client.0.trial150.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 5.7027e-05 s, 18.1 MB/s 2026-02-22T20:53:14.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: seq 1 16 2026-02-22T20:53:14.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj1 testdata.460661 2026-02-22T20:53:14.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj2 testdata.460661 2026-02-22T20:53:14.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj3 testdata.460661 2026-02-22T20:53:14.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj4 testdata.460661 2026-02-22T20:53:14.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj5 testdata.460661 2026-02-22T20:53:14.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj6 testdata.460661 2026-02-22T20:53:14.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj7 testdata.460661 2026-02-22T20:53:14.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj8 testdata.460661 2026-02-22T20:53:14.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj9 testdata.460661 2026-02-22T20:53:14.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj10 testdata.460661 2026-02-22T20:53:14.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj11 testdata.460661 2026-02-22T20:53:14.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj12 testdata.460661 2026-02-22T20:53:14.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj13 testdata.460661 2026-02-22T20:53:14.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj14 testdata.460661 2026-02-22T20:53:14.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj15 testdata.460661 2026-02-22T20:53:14.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:195: TEST_scrub_snaps: for i in `seq 1 $OBJS` 2026-02-22T20:53:14.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:197: TEST_scrub_snaps: rados -p test put obj16 testdata.460661 2026-02-22T20:53:14.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: get_primary test obj1 2026-02-22T20:53:14.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-22T20:53:14.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:53:14.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-22T20:53:14.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:53:14.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:200: TEST_scrub_snaps: local primary=0 2026-02-22T20:53:14.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:202: TEST_scrub_snaps: create_scenario td/osd-scrub-snaps test testdata.460661 0 2026-02-22T20:53:14.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2026-02-22T20:53:14.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2026-02-22T20:53:14.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.460661 2026-02-22T20:53:14.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2026-02-22T20:53:14.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2026-02-22T20:53:14.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2026-02-22T20:53:14.978 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap1 2026-02-22T20:53:14.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=1 2026-02-22T20:53:14.982 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T20:53:14.982 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T20:53:14.982 INFO:tasks.workunit.client.0.trial150.stderr:256 bytes copied, 8.7775e-05 s, 2.9 MB/s 2026-02-22T20:53:14.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.460661 2026-02-22T20:53:15.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.460661 2026-02-22T20:53:15.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.460661 2026-02-22T20:53:15.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2026-02-22T20:53:15.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:53:15.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.460661 2026-02-22T20:53:15.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:53:15.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.460661 2026-02-22T20:53:15.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:53:15.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.460661 2026-02-22T20:53:15.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:53:15.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.460661 2026-02-22T20:53:15.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:53:15.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.460661 2026-02-22T20:53:15.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:53:15.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.460661 2026-02-22T20:53:15.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:53:15.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.460661 2026-02-22T20:53:15.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:53:15.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.460661 2026-02-22T20:53:15.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:53:15.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.460661 2026-02-22T20:53:15.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2026-02-22T20:53:15.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2026-02-22T20:53:15.279 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap2 2026-02-22T20:53:15.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=2 2026-02-22T20:53:15.283 INFO:tasks.workunit.client.0.trial150.stderr:2+0 records in 2026-02-22T20:53:15.283 INFO:tasks.workunit.client.0.trial150.stderr:2+0 records out 2026-02-22T20:53:15.283 INFO:tasks.workunit.client.0.trial150.stderr:512 bytes copied, 9.6381e-05 s, 5.3 MB/s 2026-02-22T20:53:15.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.460661 2026-02-22T20:53:15.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2026-02-22T20:53:15.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2026-02-22T20:53:15.381 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap3 2026-02-22T20:53:15.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=3 2026-02-22T20:53:15.385 INFO:tasks.workunit.client.0.trial150.stderr:3+0 records in 2026-02-22T20:53:15.385 INFO:tasks.workunit.client.0.trial150.stderr:3+0 records out 2026-02-22T20:53:15.385 INFO:tasks.workunit.client.0.trial150.stderr:768 bytes copied, 9.3306e-05 s, 8.2 MB/s 2026-02-22T20:53:15.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.460661 2026-02-22T20:53:15.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2026-02-22T20:53:15.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2026-02-22T20:53:15.483 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap4 2026-02-22T20:53:15.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=4 2026-02-22T20:53:15.486 INFO:tasks.workunit.client.0.trial150.stderr:4+0 records in 2026-02-22T20:53:15.486 INFO:tasks.workunit.client.0.trial150.stderr:4+0 records out 2026-02-22T20:53:15.487 INFO:tasks.workunit.client.0.trial150.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 9.2754e-05 s, 11.0 MB/s 2026-02-22T20:53:15.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.460661 2026-02-22T20:53:15.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.460661 2026-02-22T20:53:15.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2026-02-22T20:53:15.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2026-02-22T20:53:15.585 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap5 2026-02-22T20:53:15.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2026-02-22T20:53:15.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2026-02-22T20:53:15.688 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap6 2026-02-22T20:53:15.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=6 2026-02-22T20:53:15.691 INFO:tasks.workunit.client.0.trial150.stderr:6+0 records in 2026-02-22T20:53:15.691 INFO:tasks.workunit.client.0.trial150.stderr:6+0 records out 2026-02-22T20:53:15.691 INFO:tasks.workunit.client.0.trial150.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000103114 s, 14.9 MB/s 2026-02-22T20:53:15.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.460661 2026-02-22T20:53:15.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2026-02-22T20:53:15.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2026-02-22T20:53:15.790 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap7 2026-02-22T20:53:15.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2026-02-22T20:53:15.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2026-02-22T20:53:15.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2026-02-22T20:53:15.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2026-02-22T20:53:15.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:53:15.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:53:15.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:53:15.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:53:15.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:53:16.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:53:16.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2026-02-22T20:53:16.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:16.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2026-02-22T20:53:16.516 INFO:tasks.workunit.client.0.trial150.stdout:WARNING: only removing head with clones present 2026-02-22T20:53:16.517 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-22T20:53:16.613 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-22T20:53:16.613 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2026-02-22T20:53:16.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:16.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-22T20:53:16.938 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:c52c9666:::obj5:2# 2026-02-22T20:53:17.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-22T20:53:17.049 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2026-02-22T20:53:17.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:17.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:17.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-22T20:53:17.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-22T20:53:17.469 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:17.469 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-22T20:53:17.470 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:17.470 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:53:17.470 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-22T20:53:17.470 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-22T20:53:17.470 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-22T20:53:17.470 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-22T20:53:17.470 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-22T20:53:17.470 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-22T20:53:17.470 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:17.470 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-22T20:53:17.470 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:17.471 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:17.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:53:17.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:17.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:17.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:17.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:53:17.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:53:17.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:17.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:17.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:53:17.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:53:17.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:17.472 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:17.473 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:53:17.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-22T20:53:17.473 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:53:17.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-22T20:53:17.660 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:c52c9666:::obj5:1# 2026-02-22T20:53:17.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-22T20:53:18.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-22T20:53:18.001 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:18.001 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-22T20:53:18.002 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:18.002 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:53:18.002 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-22T20:53:18.002 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-22T20:53:18.002 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-22T20:53:18.002 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-22T20:53:18.002 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-22T20:53:18.002 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-22T20:53:18.002 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:18.002 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-22T20:53:18.002 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-22T20:53:18.003 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-22T20:53:18.003 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-22T20:53:18.003 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:18.003 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:53:18.003 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:18.003 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:18.003 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:18.003 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:53:18.003 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:18.003 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:53:18.004 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:18.004 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:18.004 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:18.004 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:53:18.004 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:18.004 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:18.004 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:18.004 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:53:18.004 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:53:18.004 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:18.004 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:18.005 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:53:18.005 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:53:18.005 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:53:18.005 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:18.005 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:53:18.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-22T20:53:18.006 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:53:18.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-22T20:53:18.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-22T20:53:18.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2026-02-22T20:53:18.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:18.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=18 2026-02-22T20:53:18.189 INFO:tasks.workunit.client.0.trial150.stderr:18+0 records in 2026-02-22T20:53:18.189 INFO:tasks.workunit.client.0.trial150.stderr:18+0 records out 2026-02-22T20:53:18.189 INFO:tasks.workunit.client.0.trial150.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 9.577e-05 s, 48.1 MB/s 2026-02-22T20:53:18.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.460661 2026-02-22T20:53:18.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2026-02-22T20:53:18.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:18.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=15 2026-02-22T20:53:18.637 INFO:tasks.workunit.client.0.trial150.stderr:15+0 records in 2026-02-22T20:53:18.638 INFO:tasks.workunit.client.0.trial150.stderr:15+0 records out 2026-02-22T20:53:18.638 INFO:tasks.workunit.client.0.trial150.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000143039 s, 26.8 MB/s 2026-02-22T20:53:18.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.460661 2026-02-22T20:53:18.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2026-02-22T20:53:18.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2026-02-22T20:53:19.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:19.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-22T20:53:19.295 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:0ee9ae15:::obj4:7# 2026-02-22T20:53:19.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-22T20:53:19.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:53:19.609 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:53:19.610 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.610 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:53:19.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-22T20:53:19.610 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.610 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.610 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.610 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.610 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.610 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.610 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:53:19.610 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2026-02-22T20:53:19.610 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2026-02-22T20:53:19.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:19.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-22T20:53:19.948 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:461f8b5e:::obj16:7# 2026-02-22T20:53:20.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-22T20:53:20.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:53:20.289 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:20.290 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:53:20.290 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:53:20.290 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:53:20.290 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:53:20.290 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:53:20.290 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:53:20.290 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:53:20.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-22T20:53:20.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-22T20:53:20.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2026-02-22T20:53:20.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:20.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2026-02-22T20:53:20.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:20.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2026-02-22T20:53:20.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:20.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=7 2026-02-22T20:53:20.732 INFO:tasks.workunit.client.0.trial150.stderr:7+0 records in 2026-02-22T20:53:20.732 INFO:tasks.workunit.client.0.trial150.stderr:7+0 records out 2026-02-22T20:53:20.732 INFO:tasks.workunit.client.0.trial150.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 6.6175e-05 s, 27.1 MB/s 2026-02-22T20:53:20.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.460661 2026-02-22T20:53:21.009 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2026-02-22T20:53:21.092 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:53:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:21.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2026-02-22T20:53:21.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2026-02-22T20:53:21.491 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:53:21.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:21.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2026-02-22T20:53:21.865 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2026-02-22T20:53:21.935 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:53:22.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:22.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2026-02-22T20:53:22.289 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2026-02-22T20:53:22.368 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:53:22.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:22.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2026-02-22T20:53:22.749 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2026-02-22T20:53:22.827 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:53:22.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:22.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2026-02-22T20:53:23.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2026-02-22T20:53:23.242 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:53:23.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:23.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2026-02-22T20:53:23.613 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2026-02-22T20:53:23.691 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:53:23.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:23.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2026-02-22T20:53:24.019 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2026-02-22T20:53:24.083 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:53:24.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:24.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2026-02-22T20:53:24.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2026-02-22T20:53:24.504 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:53:24.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:24.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2026-02-22T20:53:24.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2026-02-22T20:53:24.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2026-02-22T20:53:24.939 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:53:25.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:53:25.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2026-02-22T20:53:25.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2026-02-22T20:53:25.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2026-02-22T20:53:25.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:204: TEST_scrub_snaps: rm -f testdata.460661 2026-02-22T20:53:25.331 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: expr 1 - 1 2026-02-22T20:53:25.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: seq 0 0 2026-02-22T20:53:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:206: TEST_scrub_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:53:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:208: TEST_scrub_snaps: activate_osd td/osd-scrub-snaps 0 2026-02-22T20:53:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-22T20:53:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:53:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:53:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:53:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-22T20:53:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-22T20:53:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:53:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:53:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:53:25.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-22T20:53:25.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-22T20:53:25.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:53:25.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:53:25.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-22T20:53:25.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:53:25.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:53:25.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:53:25.335 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:53:25.335 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:25.335 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:25.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:53:25.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:53:25.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:53:25.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-22T20:53:25.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-22T20:53:25.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:53:25.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:53:25.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:53:25.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:53:25.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:53:25.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:53:25.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-22T20:53:25.337 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:53:25.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:53:25.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:53:25.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2026-02-22T20:53:25.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:53:25.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:53:25.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:53:25.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:53:25.356 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:25.345+0000 7f3d6b878a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:25.357 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:25.349+0000 7f3d6b878a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:25.358 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:25.349+0000 7f3d6b878a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:25.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:53:25.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:53:25.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:53:25.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:53:25.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:53:25.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:53:25.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:53:25.501 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:53:25.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:53:25.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:53:25.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:53:25.714 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:25.705+0000 7f3d6b878a40 -1 Falling back to public interface 2026-02-22T20:53:26.336 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:26.325+0000 7f3d6b878a40 -1 osd.0 17 log_to_monitors true 2026-02-22T20:53:26.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:53:26.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:53:26.659 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:53:26.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:53:26.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:53:26.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:53:26.817 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 20 up_thru 20 down_at 18 last_clean_interval [5,17) [v2:127.0.0.1:6802/2929444255,v1:127.0.0.1:6803/2929444255] [v2:127.0.0.1:6804/2929444255,v1:127.0.0.1:6805/2929444255] exists,up a7594624-9f48-4972-bc0c-05d4b36c7e0b 2026-02-22T20:53:26.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:53:26.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:53:26.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:53:26.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:210: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 25 2026-02-22T20:53:26.882 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:53:26.882 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:53:26.882 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:53:26.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:211: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 5 2026-02-22T20:53:26.963 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:53:26.964 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-22T20:53:26.964 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:53:26.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:212: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2026-02-22T20:53:27.045 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:53:27.045 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-22T20:53:27.045 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:53:27.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:213: TEST_scrub_snaps: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2026-02-22T20:53:27.124 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:53:27.124 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-22T20:53:27.124 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:53:27.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:216: TEST_scrub_snaps: wait_for_clean 2026-02-22T20:53:27.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:53:27.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:53:27.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:53:27.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:53:27.132 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:53:27.132 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:53:27.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:53:27.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:53:27.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:53:27.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:53:27.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:53:27.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:53:27.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:53:27.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:53:27.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:53:27.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:53:27.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:53:27.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:53:27.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:53:27.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345922 2026-02-22T20:53:27.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345922 2026-02-22T20:53:27.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-85899345922' 2026-02-22T20:53:27.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:53:27.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-85899345922 2026-02-22T20:53:27.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:53:27.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:53:27.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-85899345922 2026-02-22T20:53:27.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:53:27.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345922 2026-02-22T20:53:27.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 85899345922' 2026-02-22T20:53:27.483 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 85899345922 2026-02-22T20:53:27.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:53:27.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 85899345922 2026-02-22T20:53:27.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:53:28.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:53:28.652 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:53:28.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 85899345922 2026-02-22T20:53:28.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:53:29.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:53:29.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:53:29.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345922 -lt 85899345922 2026-02-22T20:53:29.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:53:29.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:53:29.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:53:30.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:53:30.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:53:30.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:53:30.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:53:30.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:53:30.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:53:30.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:53:30.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:53:30.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:53:30.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:53:30.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:53:30.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:53:30.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:53:30.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:53:30.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:53:30.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:218: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_max 2026-02-22T20:53:30.585 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:53:30.585 INFO:tasks.workunit.client.0.trial150.stdout: "osd_shallow_scrub_chunk_max": "25" 2026-02-22T20:53:30.585 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:53:30.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:219: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_shallow_scrub_chunk_min 2026-02-22T20:53:30.669 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:53:30.669 INFO:tasks.workunit.client.0.trial150.stdout: "osd_shallow_scrub_chunk_min": "5" 2026-02-22T20:53:30.669 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:53:30.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:220: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_seconds 2026-02-22T20:53:30.743 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:53:30.743 INFO:tasks.workunit.client.0.trial150.stdout: "osd_pg_stat_report_interval_max_seconds": "1" 2026-02-22T20:53:30.743 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:53:30.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:221: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_pg_stat_report_interval_max_epochs 2026-02-22T20:53:30.818 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:53:30.818 INFO:tasks.workunit.client.0.trial150.stdout: "osd_pg_stat_report_interval_max_epochs": "1" 2026-02-22T20:53:30.818 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:53:30.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:222: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_max 2026-02-22T20:53:30.891 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:53:30.891 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_chunk_max": "15" 2026-02-22T20:53:30.891 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:53:30.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:223: TEST_scrub_snaps: ceph tell 'osd.*' config get osd_scrub_chunk_min 2026-02-22T20:53:30.971 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:53:30.972 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_chunk_min": "5" 2026-02-22T20:53:30.972 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:53:30.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:225: TEST_scrub_snaps: local pgid=1.0 2026-02-22T20:53:30.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:226: TEST_scrub_snaps: pg_scrub 1.0 2026-02-22T20:53:30.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-22T20:53:30.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-22T20:53:30.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-22T20:53:30.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:53:30.980 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:53:30.980 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:53:30.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:53:30.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:53:30.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:53:31.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:53:31.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:53:31.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:53:31.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:53:31.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:53:31.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:53:31.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-22T20:53:31.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:53:31.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:53:31.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:53:31.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=85899345924 2026-02-22T20:53:31.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 85899345924 2026-02-22T20:53:31.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-85899345924' 2026-02-22T20:53:31.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:53:31.475 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-85899345924 2026-02-22T20:53:31.475 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:53:31.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:53:31.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-85899345924 2026-02-22T20:53:31.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:53:31.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=85899345924 2026-02-22T20:53:31.479 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 85899345924 2026-02-22T20:53:31.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 85899345924' 2026-02-22T20:53:31.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:53:31.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345922 -lt 85899345924 2026-02-22T20:53:31.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:53:32.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:53:32.648 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:53:32.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345922 -lt 85899345924 2026-02-22T20:53:32.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:53:33.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:53:33.810 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:53:33.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 85899345924 -lt 85899345924 2026-02-22T20:53:33.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:53:33.971 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 1.0 loop 0 2026-02-22T20:53:33.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-22T20:53:33.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T20:53:33.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-22T20:53:33.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:53:33.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-22T20:53:33.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:53:34.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T20:53:34.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:53:34.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:53:34.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:53:34.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-22T20:53:34.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:53:34.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:53:34.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:53:34.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:53:34.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-22T20:53:11.010488+0000 2026-02-22T20:53:34.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-22T20:53:34.354 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0 on osd.0 to scrub 2026-02-22T20:53:34.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-22T20:53:11.010488+0000 2026-02-22T20:53:34.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:53:34.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:53:11.010488+0000 2026-02-22T20:53:34.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:53:34.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:53:34.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:53:34.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:53:34.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:53:34.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:53:34.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:53:34.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:53:34.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:53:11.010488+0000 '>' 2026-02-22T20:53:11.010488+0000 2026-02-22T20:53:34.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:53:35.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:53:35.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:53:35.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:53:35.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:53:35.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:53:35.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:53:35.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:53:35.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:53:11.010488+0000 '>' 2026-02-22T20:53:11.010488+0000 2026-02-22T20:53:35.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:53:36.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:53:36.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:53:36.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:53:36.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:53:36.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:53:36.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:53:36.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:53:36.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:53:11.010488+0000 '>' 2026-02-22T20:53:11.010488+0000 2026-02-22T20:53:36.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:53:37.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:53:37.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:53:37.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:53:37.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:53:37.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:53:37.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:53:37.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:53:38.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:53:34.375275+0000 '>' 2026-02-22T20:53:11.010488+0000 2026-02-22T20:53:38.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:53:38.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:38.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: wc -l 2026-02-22T20:53:38.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:230: TEST_scrub_snaps: test 2 = 2 2026-02-22T20:53:38.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:232: TEST_scrub_snaps: rados list-inconsistent-pg test 2026-02-22T20:53:38.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: jq '. | length' td/osd-scrub-snaps/json 2026-02-22T20:53:38.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:234: TEST_scrub_snaps: test 1 = 1 2026-02-22T20:53:38.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: jq -r '.[0]' td/osd-scrub-snaps/json 2026-02-22T20:53:38.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:236: TEST_scrub_snaps: test 1.0 = 1.0 2026-02-22T20:53:38.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:238: TEST_scrub_snaps: rados list-inconsistent-obj 1.0 2026-02-22T20:53:38.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: jq .inconsistents 2026-02-22T20:53:38.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:243: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-22T20:53:38.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2026-02-22T20:53:38.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:250: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-22T20:53:38.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:251: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-22T20:53:38.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-22T20:53:38.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:253: TEST_scrub_snaps: rados list-inconsistent-snapset 1.0 2026-02-22T20:53:38.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: jq .inconsistents 2026-02-22T20:53:38.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:255: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-22T20:53:38.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: jq .inconsistents td/osd-scrub-snaps/json 2026-02-22T20:53:38.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:634: TEST_scrub_snaps: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-22T20:53:38.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:635: TEST_scrub_snaps: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-22T20:53:38.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-22T20:53:38.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:636: TEST_scrub_snaps: test no = yes 2026-02-22T20:53:38.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:641: TEST_scrub_snaps: test '' = yes 2026-02-22T20:53:38.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: find td/osd-scrub-snaps 2026-02-22T20:53:38.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: grep 'osd[^/]*\.pid' 2026-02-22T20:53:38.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:646: TEST_scrub_snaps: pidfiles=td/osd-scrub-snaps/osd.0.pid 2026-02-22T20:53:38.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:647: TEST_scrub_snaps: pids= 2026-02-22T20:53:38.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:648: TEST_scrub_snaps: for pidfile in ${pidfiles} 2026-02-22T20:53:38.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: cat td/osd-scrub-snaps/osd.0.pid 2026-02-22T20:53:38.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:650: TEST_scrub_snaps: pids+='476025 ' 2026-02-22T20:53:38.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:653: TEST_scrub_snaps: ERRORS=0 2026-02-22T20:53:38.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: seq 1 7 2026-02-22T20:53:38.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-22T20:53:38.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap1 2026-02-22T20:53:38.225 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap1 2026-02-22T20:53:38.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-22T20:53:38.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap2 2026-02-22T20:53:38.327 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap2 2026-02-22T20:53:38.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-22T20:53:38.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap3 2026-02-22T20:53:38.429 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap3 2026-02-22T20:53:38.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-22T20:53:38.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap4 2026-02-22T20:53:38.533 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap4 2026-02-22T20:53:38.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-22T20:53:38.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap5 2026-02-22T20:53:38.633 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap5 2026-02-22T20:53:38.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-22T20:53:38.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap6 2026-02-22T20:53:38.735 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap6 2026-02-22T20:53:38.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:655: TEST_scrub_snaps: for i in `seq 1 7` 2026-02-22T20:53:38.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:657: TEST_scrub_snaps: rados -p test rmsnap snap7 2026-02-22T20:53:38.838 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap7 2026-02-22T20:53:38.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:659: TEST_scrub_snaps: sleep 5 2026-02-22T20:53:43.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:660: TEST_scrub_snaps: local -i loop=0 2026-02-22T20:53:43.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: ceph pg dump pgs 2026-02-22T20:53:43.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:661: TEST_scrub_snaps: grep -q snaptrim 2026-02-22T20:53:43.986 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:53:43.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: ceph pg dump pgs 2026-02-22T20:53:43.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:663: TEST_scrub_snaps: grep -q snaptrim_error 2026-02-22T20:53:44.160 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:53:44.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:665: TEST_scrub_snaps: break 2026-02-22T20:53:44.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:674: TEST_scrub_snaps: ceph pg dump pgs 2026-02-22T20:53:44.330 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:53:44.330 INFO:tasks.workunit.client.0.trial150.stdout:1.0 35 0 0 0 0 23416 0 0 60 0 60 active+clean+inconsistent+snaptrim_error 2026-02-22T20:53:38.833763+0000 23'60 29:141 [0] 0 [0] 0 17'56 2026-02-22T20:53:34.375275+0000 0'0 2026-02-22T20:53:11.010488+0000 7 0 periodic scrub scheduled @ 2026-02-23T21:17:31.002466+0000 33 0 2026-02-22T20:53:44.331 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:53:44.331 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:53:44.331 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:53:44.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:676: TEST_scrub_snaps: for pid in $pids 2026-02-22T20:53:44.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:678: TEST_scrub_snaps: kill -0 476025 2026-02-22T20:53:44.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:685: TEST_scrub_snaps: kill_daemons td/osd-scrub-snaps 2026-02-22T20:53:44.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:53:44.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:53:44.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:53:44.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:53:44.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:53:49.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:53:49.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:687: TEST_scrub_snaps: declare -a err_strings 2026-02-22T20:53:49.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:688: TEST_scrub_snaps: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' 2026-02-22T20:53:49.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:689: TEST_scrub_snaps: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' 2026-02-22T20:53:49.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:690: TEST_scrub_snaps: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:691: TEST_scrub_snaps: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:692: TEST_scrub_snaps: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:693: TEST_scrub_snaps: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:694: TEST_scrub_snaps: err_strings[6]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:695: TEST_scrub_snaps: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:696: TEST_scrub_snaps: err_strings[8]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:697: TEST_scrub_snaps: err_strings[9]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:698: TEST_scrub_snaps: err_strings[10]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:699: TEST_scrub_snaps: err_strings[11]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:700: TEST_scrub_snaps: err_strings[12]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:701: TEST_scrub_snaps: err_strings[13]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:702: TEST_scrub_snaps: err_strings[14]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:703: TEST_scrub_snaps: err_strings[15]='log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:704: TEST_scrub_snaps: err_strings[16]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:705: TEST_scrub_snaps: err_strings[17]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:706: TEST_scrub_snaps: err_strings[18]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:707: TEST_scrub_snaps: err_strings[19]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:708: TEST_scrub_snaps: err_strings[20]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:709: TEST_scrub_snaps: err_strings[21]='log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:710: TEST_scrub_snaps: err_strings[22]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj10:.* : is missing in clone_overlap' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : no '\''_'\'' attr' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:7 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*::obj5:4 : on disk size [(]4608[)] does not match object info size [(]512[)] adjusted for ondisk to [(]512[)]' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:2' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj5:head : expected clone .*:::obj5:1' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj5:head : 2 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj8:head : snaps.seq not set' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj7:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj3:head : on disk size [(]3840[)] does not match object info size [(]768[)] adjusted for ondisk to [(]768[)]' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj6:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:head : no '\''snapset'\'' attr' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:7 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj2:4 : clone ignored due to missing snapset' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj4:head : expected clone .*:::obj4:7' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]INF[]] : scrub [0-9]*[.]0 .*:::obj4:head : 1 missing clone[(]s[)]' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj1:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj9:1 : is missing in clone_size' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj11:1 : is an unexpected clone' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj14:1 : size 1032 != clone_size 1033' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : scrub [0-9]*[.]0 .*:::obj15:head : can'\''t decode '\''snapset'\'' attr ' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:712: TEST_scrub_snaps: for err_string in "${err_strings[@]}" 2026-02-22T20:53:49.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:714: TEST_scrub_snaps: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:53:49.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:721: TEST_scrub_snaps: '[' 0 '!=' 0 ']' 2026-02-22T20:53:49.704 INFO:tasks.workunit.client.0.trial150.stdout:TEST PASSED 2026-02-22T20:53:49.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:727: TEST_scrub_snaps: echo 'TEST PASSED' 2026-02-22T20:53:49.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:728: TEST_scrub_snaps: return 0 2026-02-22T20:53:49.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2026-02-22T20:53:49.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-22T20:53:49.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:53:49.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-22T20:53:49.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:53:49.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:53:49.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:53:49.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:53:49.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:53:49.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:53:49.711 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:53:49.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:53:49.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:53:49.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:53:49.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:53:49.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:53:49.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:53:49.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:53:49.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:53:49.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:53:49.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:53:49.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:53:49.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-22T20:53:49.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:53:49.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:49.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:49.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.460661 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:53:49.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:53:49.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:53:49.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:53:49.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:53:49.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:53:49.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:53:49.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:53:49.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:53:49.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:53:49.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:53:49.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:53:49.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:53:49.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:53:49.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:53:49.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-22T20:53:49.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:53:49.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:49.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:49.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.460661 2026-02-22T20:53:49.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:53:49.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:53:49.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2026-02-22T20:53:49.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:53:49.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:49.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:49.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.460661 2026-02-22T20:53:49.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:53:49.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_primary td/osd-scrub-snaps 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1177: TEST_scrub_snaps_primary: local dir=td/osd-scrub-snaps 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1178: TEST_scrub_snaps_primary: ORIG_ARGS='--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1179: TEST_scrub_snaps_primary: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1180: TEST_scrub_snaps_primary: _scrub_snaps_multi td/osd-scrub-snaps primary 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=primary 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.460661 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2026-02-22T20:53:49.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2026-02-22T20:53:49.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:53:49.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:53:49.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:53:49.772 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:53:49.772 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:49.772 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:49.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:53:49.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-22T20:53:49.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:53:49.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:53:49.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:53:49.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:53:49.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:53:49.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:53:49.806 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:53:49.807 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:53:49.807 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:53:49.807 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:53:49.807 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:49.807 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:49.807 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.460661/ceph-mon.a.asok 2026-02-22T20:53:49.808 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:53:49.808 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.460661/ceph-mon.a.asok config get fsid 2026-02-22T20:53:49.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:53:49.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:53:49.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:53:49.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:53:49.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:53:49.863 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:53:49.864 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:53:49.864 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:53:49.864 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:53:49.864 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:49.864 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:49.864 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.460661/ceph-mon.a.asok 2026-02-22T20:53:49.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:53:49.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.460661/ceph-mon.a.asok config get mon_host 2026-02-22T20:53:49.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2026-02-22T20:53:49.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2026-02-22T20:53:49.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:53:49.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:53:49.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:53:49.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2026-02-22T20:53:49.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:53:50.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:53:50.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:53:50.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:53:50.017 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:53:50.017 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:50.017 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:50.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:53:50.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:53:50.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:53:50.039 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2026-02-22T20:53:50.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2026-02-22T20:53:50.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:53:50.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2026-02-22T20:53:50.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-22T20:53:50.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:53:50.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:53:50.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:53:50.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-22T20:53:50.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-22T20:53:50.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:53:50.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:53:50.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:53:50.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-22T20:53:50.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-22T20:53:50.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:53:50.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:53:50.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-22T20:53:50.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:53:50.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:53:50.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:53:50.045 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:53:50.045 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:50.045 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:50.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:53:50.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:53:50.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:53:50.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:53:50.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:53:50.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-22T20:53:50.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-22T20:53:50.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:53:50.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:53:50.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:53:50.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:53:50.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:53:50.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:53:50.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-22T20:53:50.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:53:50.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=913632dd-270d-4366-9fca-78d8e4c19477 2026-02-22T20:53:50.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 913632dd-270d-4366-9fca-78d8e4c19477' 2026-02-22T20:53:50.047 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 913632dd-270d-4366-9fca-78d8e4c19477 2026-02-22T20:53:50.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:53:50.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDebJtpD3JDAxAAIoEW6q22wSNrYp3fNIV14g== 2026-02-22T20:53:50.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDebJtpD3JDAxAAIoEW6q22wSNrYp3fNIV14g=="}' 2026-02-22T20:53:50.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 913632dd-270d-4366-9fca-78d8e4c19477 -i td/osd-scrub-snaps/0/new.json 2026-02-22T20:53:50.154 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:53:50.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2026-02-22T20:53:50.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDebJtpD3JDAxAAIoEW6q22wSNrYp3fNIV14g== --osd-uuid 913632dd-270d-4366-9fca-78d8e4c19477 2026-02-22T20:53:50.186 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:50.177+0000 7f8137725a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:50.188 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:50.177+0000 7f8137725a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:50.189 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:50.181+0000 7f8137725a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:50.189 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:50.181+0000 7f8137725a40 -1 bdev(0x55a99a4cec00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:53:50.190 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:50.181+0000 7f8137725a40 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2026-02-22T20:53:50.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2026-02-22T20:53:50.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:53:50.997 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:53:50.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:53:50.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:53:51.206 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:53:51.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:53:51.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:53:51.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:53:51.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:53:51.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:53:51.239 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:51.229+0000 7f422b78ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:51.241 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:51.233+0000 7f422b78ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:51.242 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:51.233+0000 7f422b78ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:51.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:53:51.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:53:51.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:53:51.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:53:51.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:53:51.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:53:51.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:53:51.375 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:53:51.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:53:51.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:53:51.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:53:51.638 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:51.629+0000 7f422b78ea40 -1 Falling back to public interface 2026-02-22T20:53:52.181 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:52.173+0000 7f422b78ea40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:53:52.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:53:52.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:53:52.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:53:52.538 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:53:52.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:53:52.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:53:52.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:53:53.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:53:53.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:53:53.698 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:53:53.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:53:53.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:53:53.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:53:53.865 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1297760372,v1:127.0.0.1:6803/1297760372] [v2:127.0.0.1:6804/1297760372,v1:127.0.0.1:6805/1297760372] exists,up 913632dd-270d-4366-9fca-78d8e4c19477 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/1 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:53:53.866 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:53:53.867 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:53:53.867 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:53:53.867 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:53:53.867 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:53:53.867 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:53:53.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:53:53.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:53:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:53:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:53:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-22T20:53:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-22T20:53:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:53:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:53:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:53:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:53:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:53:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:53:53.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/1 2026-02-22T20:53:53.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:53:53.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e8f99dd3-de5f-4c85-b805-9b74ba9d5179 2026-02-22T20:53:53.869 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 e8f99dd3-de5f-4c85-b805-9b74ba9d5179 2026-02-22T20:53:53.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e8f99dd3-de5f-4c85-b805-9b74ba9d5179' 2026-02-22T20:53:53.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:53:53.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDhbJtphS5tNBAAUrlmj6fECP6tkBwriJGk9Q== 2026-02-22T20:53:53.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDhbJtphS5tNBAAUrlmj6fECP6tkBwriJGk9Q=="}' 2026-02-22T20:53:53.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e8f99dd3-de5f-4c85-b805-9b74ba9d5179 -i td/osd-scrub-snaps/1/new.json 2026-02-22T20:53:54.045 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:53:54.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/1/new.json 2026-02-22T20:53:54.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhbJtphS5tNBAAUrlmj6fECP6tkBwriJGk9Q== --osd-uuid e8f99dd3-de5f-4c85-b805-9b74ba9d5179 2026-02-22T20:53:54.087 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:54.077+0000 7efe4e3a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:54.088 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:54.081+0000 7efe4e3a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:54.089 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:54.081+0000 7efe4e3a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:54.090 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:54.081+0000 7efe4e3a3a40 -1 bdev(0x55d063c1cc00 td/osd-scrub-snaps/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:53:54.090 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:54.081+0000 7efe4e3a3a40 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2026-02-22T20:53:54.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2026-02-22T20:53:54.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:53:54.705 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:53:54.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:53:54.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:53:54.913 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:53:54.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:53:54.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:53:54.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:53:54.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:53:54.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:53:54.944 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:54.933+0000 7f630fddaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:54.946 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:54.937+0000 7f630fddaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:54.947 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:54.937+0000 7f630fddaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:53:55.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:53:55.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:53:55.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:53:55.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:53:55.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:53:55.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:53:55.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:53:55.079 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:53:55.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:53:55.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:53:55.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:53:55.346 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:55.337+0000 7f630fddaa40 -1 Falling back to public interface 2026-02-22T20:53:55.936 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:53:55.925+0000 7f630fddaa40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:53:56.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:53:56.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:53:56.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:53:56.244 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:53:56.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:53:56.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:53:56.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:53:57.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:53:57.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:53:57.406 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:53:57.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:53:57.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:53:57.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:53:57.568 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1292145473,v1:127.0.0.1:6811/1292145473] [v2:127.0.0.1:6812/1292145473,v1:127.0.0.1:6813/1292145473] exists,up e8f99dd3-de5f-4c85-b805-9b74ba9d5179 2026-02-22T20:53:57.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:53:57.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:53:57.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:53:57.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2026-02-22T20:53:57.773 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:53:57.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2026-02-22T20:53:57.977 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:53:57.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2026-02-22T20:53:57.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-22T20:53:58.181 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T20:53:58.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:53:59.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2026-02-22T20:53:59.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:53:59.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:53:59.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:53:59.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:53:59.193 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:53:59.194 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:53:59.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:53:59.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:53:59.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:53:59.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:53:59.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:53:59.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:53:59.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:53:59.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:53:59.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:53:59.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:53:59.463 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:53:59.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:53:59.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:53:59.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:53:59.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:53:59.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:53:59.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:53:59.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:53:59.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:53:59.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-22T20:53:59.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-22T20:53:59.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-22T20:53:59.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:53:59.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:53:59.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:53:59.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:53:59.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:53:59.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:53:59.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:53:59.623 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:53:59.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:53:59.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:53:59.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:53:59.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:54:00.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:54:00.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:54:00.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T20:54:00.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:54:00.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-22T20:54:00.950 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:54:00.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:54:00.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-22T20:54:00.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:54:00.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-22T20:54:00.955 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705666 2026-02-22T20:54:00.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-22T20:54:00.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:54:01.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-22T20:54:01.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:54:01.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:54:01.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:54:01.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:54:01.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:54:01.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:54:01.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:54:01.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:54:01.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:54:01.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:54:01.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:54:01.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:54:01.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:54:01.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:54:01.489 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:54:01.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:54:01.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:54:01.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:54:01.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2026-02-22T20:54:01.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2026-02-22T20:54:01.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2026-02-22T20:54:01.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2026-02-22T20:54:01.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.460661 bs=1032 count=1 2026-02-22T20:54:01.846 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T20:54:01.846 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T20:54:01.846 INFO:tasks.workunit.client.0.trial150.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.5146e-05 s, 29.4 MB/s 2026-02-22T20:54:01.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2026-02-22T20:54:01.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:01.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.460661 2026-02-22T20:54:01.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:01.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.460661 2026-02-22T20:54:01.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:01.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.460661 2026-02-22T20:54:01.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:01.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.460661 2026-02-22T20:54:01.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:01.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.460661 2026-02-22T20:54:01.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:01.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.460661 2026-02-22T20:54:02.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:02.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.460661 2026-02-22T20:54:02.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:02.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.460661 2026-02-22T20:54:02.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:02.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.460661 2026-02-22T20:54:02.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:02.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.460661 2026-02-22T20:54:02.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:02.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.460661 2026-02-22T20:54:02.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:02.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.460661 2026-02-22T20:54:02.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:02.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.460661 2026-02-22T20:54:02.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:02.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.460661 2026-02-22T20:54:02.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:02.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.460661 2026-02-22T20:54:02.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:02.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.460661 2026-02-22T20:54:02.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2026-02-22T20:54:02.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-22T20:54:02.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:54:02.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-22T20:54:02.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:54:02.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2026-02-22T20:54:02.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2026-02-22T20:54:02.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2026-02-22T20:54:02.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-22T20:54:02.461 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2026-02-22T20:54:02.461 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-22T20:54:02.461 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:54:02.461 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-22T20:54:02.461 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:54:02.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:54:02.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2026-02-22T20:54:02.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:54:02.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2026-02-22T20:54:02.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.460661 '$primary' 2026-02-22T20:54:02.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.460661 1 2026-02-22T20:54:02.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2026-02-22T20:54:02.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2026-02-22T20:54:02.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.460661 2026-02-22T20:54:02.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=1 2026-02-22T20:54:02.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2026-02-22T20:54:02.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2026-02-22T20:54:02.903 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap1 2026-02-22T20:54:02.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=1 2026-02-22T20:54:02.907 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T20:54:02.907 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T20:54:02.907 INFO:tasks.workunit.client.0.trial150.stderr:256 bytes copied, 0.000177934 s, 1.4 MB/s 2026-02-22T20:54:02.907 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.460661 2026-02-22T20:54:02.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.460661 2026-02-22T20:54:02.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.460661 2026-02-22T20:54:02.976 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2026-02-22T20:54:02.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:02.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.460661 2026-02-22T20:54:03.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:03.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.460661 2026-02-22T20:54:03.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:03.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.460661 2026-02-22T20:54:03.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:03.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.460661 2026-02-22T20:54:03.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:03.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.460661 2026-02-22T20:54:03.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:03.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.460661 2026-02-22T20:54:03.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:03.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.460661 2026-02-22T20:54:03.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:03.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.460661 2026-02-22T20:54:03.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:03.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.460661 2026-02-22T20:54:03.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2026-02-22T20:54:03.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2026-02-22T20:54:03.311 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap2 2026-02-22T20:54:03.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=2 2026-02-22T20:54:03.315 INFO:tasks.workunit.client.0.trial150.stderr:2+0 records in 2026-02-22T20:54:03.315 INFO:tasks.workunit.client.0.trial150.stderr:2+0 records out 2026-02-22T20:54:03.315 INFO:tasks.workunit.client.0.trial150.stderr:512 bytes copied, 8.6893e-05 s, 5.9 MB/s 2026-02-22T20:54:03.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.460661 2026-02-22T20:54:03.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2026-02-22T20:54:03.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2026-02-22T20:54:03.413 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap3 2026-02-22T20:54:03.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=3 2026-02-22T20:54:03.416 INFO:tasks.workunit.client.0.trial150.stderr:3+0 records in 2026-02-22T20:54:03.416 INFO:tasks.workunit.client.0.trial150.stderr:3+0 records out 2026-02-22T20:54:03.416 INFO:tasks.workunit.client.0.trial150.stderr:768 bytes copied, 9.9106e-05 s, 7.7 MB/s 2026-02-22T20:54:03.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.460661 2026-02-22T20:54:03.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2026-02-22T20:54:03.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2026-02-22T20:54:03.515 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap4 2026-02-22T20:54:03.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=4 2026-02-22T20:54:03.518 INFO:tasks.workunit.client.0.trial150.stderr:4+0 records in 2026-02-22T20:54:03.519 INFO:tasks.workunit.client.0.trial150.stderr:4+0 records out 2026-02-22T20:54:03.519 INFO:tasks.workunit.client.0.trial150.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 9.7664e-05 s, 10.5 MB/s 2026-02-22T20:54:03.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.460661 2026-02-22T20:54:03.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.460661 2026-02-22T20:54:03.566 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2026-02-22T20:54:03.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2026-02-22T20:54:03.616 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap5 2026-02-22T20:54:03.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2026-02-22T20:54:03.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2026-02-22T20:54:03.718 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap6 2026-02-22T20:54:03.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=6 2026-02-22T20:54:03.720 INFO:tasks.workunit.client.0.trial150.stderr:6+0 records in 2026-02-22T20:54:03.720 INFO:tasks.workunit.client.0.trial150.stderr:6+0 records out 2026-02-22T20:54:03.720 INFO:tasks.workunit.client.0.trial150.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000109966 s, 14.0 MB/s 2026-02-22T20:54:03.720 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.460661 2026-02-22T20:54:03.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2026-02-22T20:54:03.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2026-02-22T20:54:03.821 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap7 2026-02-22T20:54:03.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2026-02-22T20:54:03.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2026-02-22T20:54:03.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2026-02-22T20:54:03.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2026-02-22T20:54:03.905 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:54:03.905 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:54:03.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:54:03.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:54:03.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:54:04.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:54:04.213 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj1 2026-02-22T20:54:04.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:04.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2026-02-22T20:54:04.581 INFO:tasks.workunit.client.0.trial150.stdout:WARNING: only removing head with clones present 2026-02-22T20:54:04.581 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-22T20:54:04.694 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2026-02-22T20:54:04.694 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2026-02-22T20:54:04.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:04.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-22T20:54:05.027 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:c52c9666:::obj5:2# 2026-02-22T20:54:05.118 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2026-02-22T20:54:05.118 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2026-02-22T20:54:05.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:05.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:05.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2026-02-22T20:54:05.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:05.497 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:05.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-22T20:54:05.499 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:54:05.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-22T20:54:05.672 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:c52c9666:::obj5:1# 2026-02-22T20:54:05.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2026-02-22T20:54:05.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.969 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:05.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-22T20:54:05.971 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:54:05.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-22T20:54:05.972 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj5 2026-02-22T20:54:05.972 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2026-02-22T20:54:06.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:06.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=18 2026-02-22T20:54:06.118 INFO:tasks.workunit.client.0.trial150.stderr:18+0 records in 2026-02-22T20:54:06.118 INFO:tasks.workunit.client.0.trial150.stderr:18+0 records out 2026-02-22T20:54:06.118 INFO:tasks.workunit.client.0.trial150.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.000149852 s, 30.8 MB/s 2026-02-22T20:54:06.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.460661 2026-02-22T20:54:06.396 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj3 2026-02-22T20:54:06.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:06.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=15 2026-02-22T20:54:06.573 INFO:tasks.workunit.client.0.trial150.stderr:15+0 records in 2026-02-22T20:54:06.573 INFO:tasks.workunit.client.0.trial150.stderr:15+0 records out 2026-02-22T20:54:06.574 INFO:tasks.workunit.client.0.trial150.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000199675 s, 19.2 MB/s 2026-02-22T20:54:06.574 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.460661 2026-02-22T20:54:06.854 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj4 2026-02-22T20:54:06.854 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2026-02-22T20:54:07.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:07.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-22T20:54:07.204 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:0ee9ae15:::obj4:7# 2026-02-22T20:54:07.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2026-02-22T20:54:07.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-22T20:54:07.529 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-22T20:54:07.529 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.529 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:54:07.529 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-22T20:54:07.529 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-22T20:54:07.529 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-22T20:54:07.530 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-22T20:54:07.530 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-22T20:54:07.530 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-22T20:54:07.530 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:07.530 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-22T20:54:07.530 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-22T20:54:07.530 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-22T20:54:07.530 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-22T20:54:07.530 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:07.530 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.530 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:07.530 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:07.531 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.531 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:07.531 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:07.531 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.531 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:07.531 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:07.531 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.531 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:07.531 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:07.531 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.531 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:07.531 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:07.532 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.532 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:07.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-22T20:54:07.533 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.533 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.533 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.533 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.533 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.533 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.533 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:07.534 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --op list obj16 2026-02-22T20:54:07.534 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2026-02-22T20:54:07.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:07.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-22T20:54:07.884 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:461f8b5e:::obj16:7# 2026-02-22T20:54:07.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/1 list 2026-02-22T20:54:08.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:08.173 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:08.174 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:08.174 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:08.174 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:08.174 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:08.174 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:08.174 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:08.174 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:08.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-22T20:54:08.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-22T20:54:08.175 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj2 2026-02-22T20:54:08.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:08.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2026-02-22T20:54:08.625 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:08.625 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2026-02-22T20:54:08.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:08.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=7 2026-02-22T20:54:08.627 INFO:tasks.workunit.client.0.trial150.stderr:7+0 records in 2026-02-22T20:54:08.628 INFO:tasks.workunit.client.0.trial150.stderr:7+0 records out 2026-02-22T20:54:08.628 INFO:tasks.workunit.client.0.trial150.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 6.5553e-05 s, 27.3 MB/s 2026-02-22T20:54:08.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.460661 2026-02-22T20:54:08.925 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj6 2026-02-22T20:54:08.991 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:09.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:09.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2026-02-22T20:54:09.394 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj7 2026-02-22T20:54:09.463 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:09.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:09.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2026-02-22T20:54:09.829 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj8 2026-02-22T20:54:09.927 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:10.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:10.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2026-02-22T20:54:10.290 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj9 2026-02-22T20:54:10.371 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:10.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:10.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2026-02-22T20:54:10.725 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj10 2026-02-22T20:54:10.808 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:10.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:10.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2026-02-22T20:54:11.169 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj11 2026-02-22T20:54:11.243 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:11.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:11.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2026-02-22T20:54:11.625 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj12 2026-02-22T20:54:11.703 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:11.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:11.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2026-02-22T20:54:12.089 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj13 2026-02-22T20:54:12.172 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:12.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:12.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2026-02-22T20:54:12.525 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj14 2026-02-22T20:54:12.603 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:12.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:12.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2026-02-22T20:54:12.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2026-02-22T20:54:12.946 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 --head --op list obj15 2026-02-22T20:54:13.020 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:13.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:13.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/1 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2026-02-22T20:54:13.388 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2026-02-22T20:54:13.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2026-02-22T20:54:13.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.460661 2026-02-22T20:54:13.391 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2026-02-22T20:54:13.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2026-02-22T20:54:13.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:54:13.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2026-02-22T20:54:13.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-22T20:54:13.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:54:13.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:54:13.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:54:13.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-22T20:54:13.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-22T20:54:13.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:54:13.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:54:13.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:54:13.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-22T20:54:13.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-22T20:54:13.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:54:13.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:54:13.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-22T20:54:13.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:54:13.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:54:13.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:54:13.398 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:54:13.398 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:54:13.398 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:54:13.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:54:13.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:54:13.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:54:13.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-22T20:54:13.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-22T20:54:13.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:54:13.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:54:13.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:54:13.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:54:13.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:54:13.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:54:13.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-22T20:54:13.400 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:54:13.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:54:13.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:54:13.401 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2026-02-22T20:54:13.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:54:13.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:54:13.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:54:13.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:54:13.425 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:13.417+0000 7fdc1a49aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:13.426 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:13.417+0000 7fdc1a49aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:13.427 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:13.417+0000 7fdc1a49aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:13.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:54:13.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:54:13.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:54:13.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:54:13.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:54:13.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:54:13.566 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:54:13.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:54:13.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:54:13.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:54:13.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:54:13.762 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:13.753+0000 7fdc1a49aa40 -1 Falling back to public interface 2026-02-22T20:54:14.370 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:14.361+0000 7fdc1a49aa40 -1 osd.0 21 log_to_monitors true 2026-02-22T20:54:14.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:54:14.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:54:14.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:54:14.729 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:54:14.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:54:14.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:54:14.889 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 24 up_thru 24 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/498244337,v1:127.0.0.1:6803/498244337] [v2:127.0.0.1:6804/498244337,v1:127.0.0.1:6805/498244337] exists,up 913632dd-270d-4366-9fca-78d8e4c19477 2026-02-22T20:54:14.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:54:14.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:54:14.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:54:14.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:54:14.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2026-02-22T20:54:14.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-22T20:54:14.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:54:14.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:54:14.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:54:14.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/1 2026-02-22T20:54:14.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-22T20:54:14.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:54:14.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:54:14.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:54:14.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2026-02-22T20:54:14.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2026-02-22T20:54:14.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:54:14.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:54:14.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:54:14.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:54:14.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:54:14.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:54:14.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:54:14.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:54:14.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/1 2026-02-22T20:54:14.896 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:54:14.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:54:14.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:54:14.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/1/whoami 2026-02-22T20:54:14.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:54:14.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:54:14.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:54:14.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:54:14.919 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:14.909+0000 7fe6043e0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:14.920 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:14.913+0000 7fe6043e0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:14.921 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:14.913+0000 7fe6043e0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:15.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:54:15.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:54:15.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:54:15.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:54:15.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:54:15.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:54:15.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:54:15.057 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:54:15.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:54:15.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:54:15.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:54:15.246 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:15.237+0000 7fe6043e0a40 -1 Falling back to public interface 2026-02-22T20:54:15.844 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:15.833+0000 7fe6043e0a40 -1 osd.1 21 log_to_monitors true 2026-02-22T20:54:16.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:54:16.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:54:16.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:54:16.222 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:54:16.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:54:16.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:54:16.388 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 28 up_thru 28 down_at 22 last_clean_interval [9,21) [v2:127.0.0.1:6810/3579005130,v1:127.0.0.1:6811/3579005130] [v2:127.0.0.1:6812/3579005130,v1:127.0.0.1:6813/3579005130] exists,up e8f99dd3-de5f-4c85-b805-9b74ba9d5179 2026-02-22T20:54:16.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:54:16.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:54:16.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:54:16.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2026-02-22T20:54:16.458 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:54:16.458 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:54:16.458 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:54:16.465 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:54:16.466 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:54:16.466 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:54:16.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2026-02-22T20:54:16.548 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:54:16.548 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-22T20:54:16.548 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:54:16.555 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:54:16.555 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-22T20:54:16.555 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:54:16.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2026-02-22T20:54:16.635 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:54:16.636 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_min = '' " 2026-02-22T20:54:16.636 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:54:16.643 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:54:16.643 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_min = '' " 2026-02-22T20:54:16.643 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:54:16.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2026-02-22T20:54:16.730 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:54:16.730 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-22T20:54:16.730 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:54:16.737 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:54:16.737 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-22T20:54:16.737 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:54:16.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2026-02-22T20:54:16.823 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:54:16.823 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-22T20:54:16.823 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:54:16.830 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:54:16.830 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-22T20:54:16.831 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:54:16.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2026-02-22T20:54:16.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:54:16.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:54:16.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:54:16.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:54:16.840 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:54:16.840 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:54:16.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:54:16.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:54:16.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:54:16.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:54:16.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:54:16.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:54:16.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:54:16.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:54:16.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:54:17.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:54:17.112 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:54:17.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:54:17.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:54:17.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:54:17.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-22T20:54:17.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-22T20:54:17.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-22T20:54:17.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:54:17.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:54:17.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084290 2026-02-22T20:54:17.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084290 2026-02-22T20:54:17.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-120259084290' 2026-02-22T20:54:17.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:54:17.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-22T20:54:17.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:54:17.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:54:17.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-22T20:54:17.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:54:17.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-22T20:54:17.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-22T20:54:17.273 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 103079215106 2026-02-22T20:54:17.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:54:17.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-22T20:54:17.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:54:18.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:54:18.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:54:18.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215106 2026-02-22T20:54:18.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:54:18.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084290 2026-02-22T20:54:18.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:54:18.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:54:18.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084290 2026-02-22T20:54:18.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:54:18.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084290 2026-02-22T20:54:18.618 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 120259084290 2026-02-22T20:54:18.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084290' 2026-02-22T20:54:18.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:54:18.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084290 -lt 120259084290 2026-02-22T20:54:18.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:54:18.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:54:18.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:54:18.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:54:18.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:54:18.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:54:18.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:54:18.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:54:18.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:54:18.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:54:18.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:54:19.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:54:19.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:54:19.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:54:19.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:54:19.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:54:19.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:54:19.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:54:19.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2026-02-22T20:54:19.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2026-02-22T20:54:19.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-22T20:54:19.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-22T20:54:19.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-22T20:54:19.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:54:19.346 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:54:19.346 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:54:19.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:54:19.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:54:19.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:54:19.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:54:19.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:54:19.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:54:19.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:54:19.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:54:19.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:54:19.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:54:19.789 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:54:19.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:54:19.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:54:19.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:54:19.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215108 2026-02-22T20:54:19.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215108 2026-02-22T20:54:19.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108' 2026-02-22T20:54:19.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:54:19.866 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:54:19.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084291 2026-02-22T20:54:19.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084291 2026-02-22T20:54:19.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108 1-120259084291' 2026-02-22T20:54:19.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:54:19.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215108 2026-02-22T20:54:19.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:54:19.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:54:19.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215108 2026-02-22T20:54:19.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:54:19.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215108 2026-02-22T20:54:19.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2026-02-22T20:54:19.944 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 103079215108 2026-02-22T20:54:19.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:54:20.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215108 2026-02-22T20:54:20.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:54:21.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:54:21.107 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:54:21.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215108 -lt 103079215108 2026-02-22T20:54:21.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:54:21.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084291 2026-02-22T20:54:21.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:54:21.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:54:21.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084291 2026-02-22T20:54:21.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:54:21.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084291 2026-02-22T20:54:21.274 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 120259084291 2026-02-22T20:54:21.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084291' 2026-02-22T20:54:21.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:54:21.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084291 -lt 120259084291 2026-02-22T20:54:21.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:54:21.436 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 1.0 loop 0 2026-02-22T20:54:21.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-22T20:54:21.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T20:54:21.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-22T20:54:21.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:54:21.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-22T20:54:21.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:54:21.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T20:54:21.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:54:21.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:54:21.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:54:21.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-22T20:54:21.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:54:21.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:54:21.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:54:21.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:54:21.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-22T20:53:58.173700+0000 2026-02-22T20:54:21.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-22T20:54:21.827 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0 on osd.1 to scrub 2026-02-22T20:54:21.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-22T20:53:58.173700+0000 2026-02-22T20:54:21.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:54:21.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:53:58.173700+0000 2026-02-22T20:54:21.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:54:21.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:54:21.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:54:21.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:54:21.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:54:21.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:54:21.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:54:21.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:54:22.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:53:58.173700+0000 '>' 2026-02-22T20:53:58.173700+0000 2026-02-22T20:54:22.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:54:23.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:54:23.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:54:23.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:54:23.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:54:23.004 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:54:23.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:54:23.005 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:54:23.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:53:58.173700+0000 '>' 2026-02-22T20:53:58.173700+0000 2026-02-22T20:54:23.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:54:24.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:54:24.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:54:24.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:54:24.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:54:24.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:54:24.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:54:24.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:54:24.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:53:58.173700+0000 '>' 2026-02-22T20:53:58.173700+0000 2026-02-22T20:54:24.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:54:25.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:54:25.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:54:25.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:54:25.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:54:25.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:54:25.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:54:25.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:54:25.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:53:58.173700+0000 '>' 2026-02-22T20:53:58.173700+0000 2026-02-22T20:54:25.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:54:26.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:54:26.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:54:26.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:54:26.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:54:26.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:54:26.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:54:26.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:54:26.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:54:21.836324+0000 '>' 2026-02-22T20:53:58.173700+0000 2026-02-22T20:54:26.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:54:26.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:54:26.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2026-02-22T20:54:26.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 14 -gt 3 2026-02-22T20:54:26.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:54:26.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2026-02-22T20:54:26.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 14 -gt 3 2026-02-22T20:54:26.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2026-02-22T20:54:26.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2026-02-22T20:54:26.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2026-02-22T20:54:26.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2026-02-22T20:54:26.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2026-02-22T20:54:26.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "epoch": 28, 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "inconsistents": [ 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj1", 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "version": 18 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [ 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "missing" 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj1", 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 1828249343, 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'18", 2026-02-22T20:54:26.731 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'1", 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4222.0:1", 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 18, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:02.920390+0000", 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:02.921844+0000", 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0x78aeffa4", 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: "missing" 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.732 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj10", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "version": 32 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj10", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 718195851, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'32", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'10", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4243.0:1", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 32, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:03.110773+0000", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:03.111115+0000", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0x78aeffa4", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.733 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.734 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "????", 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj11", 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "version": 34 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj11", 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 693400951, 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'34", 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'11", 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4246.0:1", 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 34, 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:03.139669+0000", 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:03.139952+0000", 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.735 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0x78aeffa4", 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.736 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [] 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj13", 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "version": 38 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj13", 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 2087409765, 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'38", 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'13", 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4252.0:1", 2026-02-22T20:54:26.737 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 38, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:03.197750+0000", 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:03.198143+0000", 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0x78aeffa4", 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.738 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]" 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj14", 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "version": 40 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj14", 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 2484217095, 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.739 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'40", 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'14", 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4255.0:1", 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 40, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:03.226925+0000", 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:03.227310+0000", 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0x78aeffa4", 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.740 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1033, 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj15", 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "version": 15 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.741 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [ 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_corrupted" 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj15", 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 612772309, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "version": "14'15", 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "0'0", 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4207.0:1", 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 15, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:02.257860+0000", 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:02.258177+0000", 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xb6993e72", 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:54:26.742 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 0, 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [] 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_corrupted" 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": "Z2FyYmFnZQo=" 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj2", 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "version": 56 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [ 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_missing" 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj2", 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 1058988552, 2026-02-22T20:54:26.743 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "version": "21'56", 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "18'48", 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4297.0:1", 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 56, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "size": 0, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:03.896032+0000", 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:03.896432+0000", 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "whiteout", 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "dirty" 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xffffffff", 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "size": 0, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 7, 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.744 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 4, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: 4, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: 3, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: 2, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 7, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1024, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: 7, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: 6, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: 5 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_missing" 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "size": 0 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj3", 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:54:26.745 INFO:tasks.workunit.client.0.trial150.stdout: "version": 44 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "size_mismatch" 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [ 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "size_mismatch_info", 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "obj_size_info_mismatch" 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj3", 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 1643547569, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "version": "17'44", 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "15'22", 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4267.0:1", 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 44, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "size": 768, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:03.438434+0000", 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:03.439565+0000", 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0x625b62a1", 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.746 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "size": 768 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "size_mismatch_info", 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "obj_size_info_mismatch" 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "size": 3840, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "object_info": { 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj3", 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 1643547569, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "version": "17'44", 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "15'22", 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4267.0:1", 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 44, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "size": 768, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:03.438434+0000", 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:03.439565+0000", 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.747 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0x625b62a1", 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj6", 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "version": 24 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj6", 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 2202164420, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'24", 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'6", 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4231.0:1", 2026-02-22T20:54:26.748 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 24, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:02.992456+0000", 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:02.992749+0000", 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0x78aeffa4", 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.749 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [] 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj7", 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "version": 26 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj7", 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 1552453721, 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'26", 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'7", 2026-02-22T20:54:26.750 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4234.0:1", 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 26, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:03.022439+0000", 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:03.022752+0000", 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0x78aeffa4", 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.751 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 0, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [] 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj8", 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "version": 28 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj8", 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 2381834917, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'28", 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'8", 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4237.0:1", 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 28, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.752 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:03.051296+0000", 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:03.051698+0000", 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0x78aeffa4", 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.753 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 0, 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:54:26.754 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj9", 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "version": 30 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj9", 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 3833113727, 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'30", 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'9", 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4240.0:1", 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 30, 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.755 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:03.081954+0000", 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:03.082214+0000", 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0x78aeffa4", 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.756 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: "size": "????", 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:54:26.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2026-02-22T20:54:26.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' primary = replica ']' 2026-02-22T20:54:26.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:811: _scrub_snaps_multi: scruberrors=30 2026-02-22T20:54:26.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: jq .inconsistents 2026-02-22T20:54:26.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:812: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-22T20:54:26.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2026-02-22T20:54:26.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-22T20:54:26.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-22T20:54:26.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-22T20:54:26.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2026-02-22T20:54:26.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2026-02-22T20:54:26.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2026-02-22T20:54:26.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2026-02-22T20:54:26.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2026-02-22T20:54:26.792 INFO:tasks.workunit.client.0.trial150.stderr:td/osd-scrub-snaps/osd.0.pid' 2026-02-22T20:54:26.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2026-02-22T20:54:26.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2026-02-22T20:54:26.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2026-02-22T20:54:26.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='495464 ' 2026-02-22T20:54:26.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2026-02-22T20:54:26.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2026-02-22T20:54:26.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='494922 ' 2026-02-22T20:54:26.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2026-02-22T20:54:26.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' primary = primary ']' 2026-02-22T20:54:26.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: seq 1 7 2026-02-22T20:54:26.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-22T20:54:26.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap1 2026-02-22T20:54:26.869 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap1 2026-02-22T20:54:26.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-22T20:54:26.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap2 2026-02-22T20:54:26.971 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap2 2026-02-22T20:54:26.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-22T20:54:26.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap3 2026-02-22T20:54:27.072 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap3 2026-02-22T20:54:27.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-22T20:54:27.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap4 2026-02-22T20:54:27.173 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap4 2026-02-22T20:54:27.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-22T20:54:27.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap5 2026-02-22T20:54:27.277 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap5 2026-02-22T20:54:27.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-22T20:54:27.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap6 2026-02-22T20:54:27.378 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap6 2026-02-22T20:54:27.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1091: _scrub_snaps_multi: for i in `seq 1 7` 2026-02-22T20:54:27.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1093: _scrub_snaps_multi: rados -p test rmsnap snap7 2026-02-22T20:54:27.480 INFO:tasks.workunit.client.0.trial150.stdout:removed pool test snap snap7 2026-02-22T20:54:27.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1095: _scrub_snaps_multi: sleep 5 2026-02-22T20:54:32.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1096: _scrub_snaps_multi: local -i loop=0 2026-02-22T20:54:32.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: ceph pg dump pgs 2026-02-22T20:54:32.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1097: _scrub_snaps_multi: grep -q snaptrim 2026-02-22T20:54:32.634 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:54:32.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: ceph pg dump pgs 2026-02-22T20:54:32.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1099: _scrub_snaps_multi: grep -q snaptrim_error 2026-02-22T20:54:32.791 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:54:32.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1101: _scrub_snaps_multi: break 2026-02-22T20:54:32.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2026-02-22T20:54:32.947 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:54:32.947 INFO:tasks.workunit.client.0.trial150.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent+snaptrim_error 2026-02-22T20:54:27.475036+0000 21'56 37:136 [1,0] 1 [1,0] 1 21'56 2026-02-22T20:54:21.836324+0000 0'0 2026-02-22T20:53:58.173700+0000 7 1 periodic scrub scheduled @ 2026-02-24T00:11:00.845475+0000 33 0 2026-02-22T20:54:32.947 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:54:32.948 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:54:32.948 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:54:32.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2026-02-22T20:54:32.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 495464 2026-02-22T20:54:32.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2026-02-22T20:54:32.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 494922 2026-02-22T20:54:32.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2026-02-22T20:54:32.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:54:32.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:54:32.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:54:32.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:54:32.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:54:38.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:54:38.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2026-02-22T20:54:38.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2026-02-22T20:54:38.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2026-02-22T20:54:38.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2026-02-22T20:54:38.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2026-02-22T20:54:38.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2026-02-22T20:54:38.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2026-02-22T20:54:38.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2026-02-22T20:54:38.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' 2026-02-22T20:54:38.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:54:38.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:54:38.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:54:38.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:54:38.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:54:38.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:54:38.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:54:38.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:54:38.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:54:38.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:54:38.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:54:38.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:54:38.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:54:38.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:54:38.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:54:38.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 30 errors' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:54:38.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2026-02-22T20:54:38.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$primary' 2026-02-22T20:54:38.293 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 1 2026-02-22T20:54:38.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=1 2026-02-22T20:54:38.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2026-02-22T20:54:38.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2026-02-22T20:54:38.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:54:38.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2026-02-22T20:54:38.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2026-02-22T20:54:38.297 INFO:tasks.workunit.client.0.trial150.stdout:TEST PASSED 2026-02-22T20:54:38.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2026-02-22T20:54:38.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1181: TEST_scrub_snaps_primary: err=0 2026-02-22T20:54:38.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1182: TEST_scrub_snaps_primary: CEPH_ARGS='--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-22T20:54:38.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1183: TEST_scrub_snaps_primary: return 0 2026-02-22T20:54:38.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2026-02-22T20:54:38.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-22T20:54:38.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:54:38.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-22T20:54:38.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:54:38.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:54:38.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:54:38.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:54:38.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:54:38.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:54:38.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:54:38.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:54:38.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:54:38.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:54:38.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:54:38.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:54:38.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:54:38.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:54:38.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:54:38.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:54:38.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:54:38.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:54:38.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-22T20:54:38.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:54:38.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:54:38.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:54:38.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.460661 2026-02-22T20:54:38.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:54:38.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:54:38.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:37: run: for func in $funcs 2026-02-22T20:54:38.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:38: run: setup td/osd-scrub-snaps 2026-02-22T20:54:38.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-snaps 2026-02-22T20:54:38.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-snaps 2026-02-22T20:54:38.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-22T20:54:38.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:54:38.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-22T20:54:38.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:54:38.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:54:38.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:54:38.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:54:38.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:54:38.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:54:38.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:54:38.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:54:38.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:54:38.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:54:38.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:54:38.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:54:38.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:54:38.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:54:38.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:54:38.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:54:38.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:54:38.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:54:38.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-22T20:54:38.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:54:38.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:54:38.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:54:38.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.460661 2026-02-22T20:54:38.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:54:38.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:54:38.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-snaps 2026-02-22T20:54:38.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:54:38.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:54:38.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:54:38.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.460661 2026-02-22T20:54:38.341 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-snaps 1' TERM HUP INT 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:39: run: TEST_scrub_snaps_replica td/osd-scrub-snaps 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1167: TEST_scrub_snaps_replica: local dir=td/osd-scrub-snaps 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1168: TEST_scrub_snaps_replica: ORIG_ARGS='--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1169: TEST_scrub_snaps_replica: CEPH_ARGS+=' --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1170: TEST_scrub_snaps_replica: _scrub_snaps_multi td/osd-scrub-snaps replica 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:732: _scrub_snaps_multi: local dir=td/osd-scrub-snaps 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:733: _scrub_snaps_multi: local poolname=test 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:734: _scrub_snaps_multi: local OBJS=16 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:735: _scrub_snaps_multi: local OSDS=2 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:736: _scrub_snaps_multi: local which=replica 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:738: _scrub_snaps_multi: TESTDATA=testdata.460661 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:740: _scrub_snaps_multi: run_mon td/osd-scrub-snaps a --osd_pool_default_size=2 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-snaps 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-snaps/a 2026-02-22T20:54:38.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-snaps/a --run-dir=td/osd-scrub-snaps --osd_pool_default_size=2 2026-02-22T20:54:38.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:54:38.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:54:38.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:54:38.366 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:54:38.366 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:54:38.366 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:54:38.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:54:38.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-snaps/a '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-snaps/log --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-22T20:54:38.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:54:38.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:54:38.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:54:38.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:54:38.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:54:38.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:54:38.400 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:54:38.400 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:54:38.400 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:54:38.400 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:54:38.400 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:54:38.400 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:54:38.400 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.460661/ceph-mon.a.asok 2026-02-22T20:54:38.401 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:54:38.401 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.460661/ceph-mon.a.asok config get fsid 2026-02-22T20:54:38.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:54:38.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:54:38.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:54:38.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:54:38.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:54:38.456 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:54:38.456 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:54:38.457 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:54:38.457 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:54:38.458 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:54:38.458 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:54:38.458 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.460661/ceph-mon.a.asok 2026-02-22T20:54:38.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:54:38.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.460661/ceph-mon.a.asok config get mon_host 2026-02-22T20:54:38.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:741: _scrub_snaps_multi: run_mgr td/osd-scrub-snaps x 2026-02-22T20:54:38.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-snaps 2026-02-22T20:54:38.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:54:38.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:54:38.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:54:38.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-snaps/x 2026-02-22T20:54:38.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:54:38.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:54:38.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:54:38.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:54:38.608 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:54:38.608 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:54:38.608 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:54:38.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:54:38.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:54:38.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-snaps/x '--log-file=td/osd-scrub-snaps/$name.log' '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --run-dir=td/osd-scrub-snaps '--pid-file=td/osd-scrub-snaps/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:54:38.631 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: expr 2 - 1 2026-02-22T20:54:38.631 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: seq 0 1 2026-02-22T20:54:38.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:54:38.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 0 2026-02-22T20:54:38.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-22T20:54:38.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:54:38.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:54:38.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:54:38.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-22T20:54:38.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:54:38.634 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:54:38.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:54:38.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:54:38.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:54:38.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:54:38.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:54:38.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-22T20:54:38.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-22T20:54:38.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:54:38.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:54:38.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:54:38.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:54:38.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:54:38.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:54:38.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-22T20:54:38.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:54:38.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2efe00df-e546-4390-961e-5210e1a63256 2026-02-22T20:54:38.637 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 2efe00df-e546-4390-961e-5210e1a63256 2026-02-22T20:54:38.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2efe00df-e546-4390-961e-5210e1a63256' 2026-02-22T20:54:38.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:54:38.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAObZtp1NJxJhAA1K6XA5dO3UrYqggMqe/iQg== 2026-02-22T20:54:38.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAObZtp1NJxJhAA1K6XA5dO3UrYqggMqe/iQg=="}' 2026-02-22T20:54:38.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2efe00df-e546-4390-961e-5210e1a63256 -i td/osd-scrub-snaps/0/new.json 2026-02-22T20:54:38.742 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:54:38.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/0/new.json 2026-02-22T20:54:38.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAObZtp1NJxJhAA1K6XA5dO3UrYqggMqe/iQg== --osd-uuid 2efe00df-e546-4390-961e-5210e1a63256 2026-02-22T20:54:38.772 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:38.761+0000 7f0a1b9fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:38.773 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:38.765+0000 7f0a1b9fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:38.775 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:38.765+0000 7f0a1b9fca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:38.775 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:38.765+0000 7f0a1b9fca40 -1 bdev(0x5623d4cf2c00 td/osd-scrub-snaps/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:54:38.775 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:38.765+0000 7f0a1b9fca40 -1 bluestore(td/osd-scrub-snaps/0) _read_fsid unparsable uuid 2026-02-22T20:54:39.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/0/keyring 2026-02-22T20:54:39.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:54:39.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:54:39.528 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:54:39.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:54:39.740 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:54:39.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:54:39.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:54:39.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:54:39.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:54:39.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:54:39.760 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:39.749+0000 7f2a71439a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:39.761 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:39.753+0000 7f2a71439a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:39.762 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:39.753+0000 7f2a71439a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:39.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:54:39.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:54:39.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:54:39.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:54:39.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:54:39.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:54:39.905 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:54:39.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:54:39.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:54:39.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:54:40.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:54:40.166 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:40.157+0000 7f2a71439a40 -1 Falling back to public interface 2026-02-22T20:54:40.811 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:40.801+0000 7f2a71439a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:54:41.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:54:41.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:54:41.074 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:54:41.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:54:41.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:54:41.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:54:41.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:54:42.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:54:42.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:54:42.239 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:54:42.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:54:42.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:54:42.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:54:42.396 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/76943547,v1:127.0.0.1:6803/76943547] [v2:127.0.0.1:6804/76943547,v1:127.0.0.1:6805/76943547] exists,up 2efe00df-e546-4390-961e-5210e1a63256 2026-02-22T20:54:42.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:54:42.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:54:42.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:54:42.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:742: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:744: _scrub_snaps_multi: run_osd td/osd-scrub-snaps 1 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-snaps 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-snaps/1 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:54:42.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T20:54:42.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-snaps/1 2026-02-22T20:54:42.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:54:42.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=56b865e5-5595-402d-af0e-2f659bcb2e2c 2026-02-22T20:54:42.400 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 56b865e5-5595-402d-af0e-2f659bcb2e2c 2026-02-22T20:54:42.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 56b865e5-5595-402d-af0e-2f659bcb2e2c' 2026-02-22T20:54:42.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:54:42.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQASbZtpsXxwGBAA6lTxnzNNI2P9UrhdQigRpw== 2026-02-22T20:54:42.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQASbZtpsXxwGBAA6lTxnzNNI2P9UrhdQigRpw=="}' 2026-02-22T20:54:42.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 56b865e5-5595-402d-af0e-2f659bcb2e2c -i td/osd-scrub-snaps/1/new.json 2026-02-22T20:54:42.581 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:54:42.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-snaps/1/new.json 2026-02-22T20:54:42.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASbZtpsXxwGBAA6lTxnzNNI2P9UrhdQigRpw== --osd-uuid 56b865e5-5595-402d-af0e-2f659bcb2e2c 2026-02-22T20:54:42.611 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:42.601+0000 7f50e9759a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:42.613 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:42.605+0000 7f50e9759a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:42.614 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:42.605+0000 7f50e9759a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:42.614 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:42.605+0000 7f50e9759a40 -1 bdev(0x557d623fec00 td/osd-scrub-snaps/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:54:42.614 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:42.605+0000 7f50e9759a40 -1 bluestore(td/osd-scrub-snaps/1) _read_fsid unparsable uuid 2026-02-22T20:54:43.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-snaps/1/keyring 2026-02-22T20:54:43.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:54:43.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:54:43.281 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:54:43.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-snaps/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:54:43.480 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:54:43.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:54:43.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:54:43.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:54:43.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:54:43.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:54:43.512 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:43.501+0000 7f8bb3883a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:43.514 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:43.505+0000 7f8bb3883a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:43.515 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:43.505+0000 7f8bb3883a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:54:43.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:54:43.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:54:43.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:54:43.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:54:43.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:54:43.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:54:43.644 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:54:43.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:54:43.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:54:43.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:54:43.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:54:43.914 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:43.905+0000 7f8bb3883a40 -1 Falling back to public interface 2026-02-22T20:54:44.451 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:54:44.441+0000 7f8bb3883a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:54:44.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:54:44.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:54:44.807 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:54:44.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:54:44.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:54:44.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:54:44.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:54:45.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:54:45.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:54:45.968 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:54:45.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T20:54:45.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:54:45.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:54:46.126 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2899317645,v1:127.0.0.1:6811/2899317645] [v2:127.0.0.1:6812/2899317645,v1:127.0.0.1:6813/2899317645] exists,up 56b865e5-5595-402d-af0e-2f659bcb2e2c 2026-02-22T20:54:46.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:54:46.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:54:46.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:54:46.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:748: _scrub_snaps_multi: ceph osd set noscrub 2026-02-22T20:54:46.332 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:54:46.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:749: _scrub_snaps_multi: ceph osd set nodeep-scrub 2026-02-22T20:54:46.537 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:54:46.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:752: _scrub_snaps_multi: create_pool test 1 1 2026-02-22T20:54:46.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-22T20:54:46.741 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T20:54:46.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:54:47.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:753: _scrub_snaps_multi: wait_for_clean 2026-02-22T20:54:47.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:54:47.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:54:47.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:54:47.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:54:47.754 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:54:47.754 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:54:47.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:54:47.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:54:47.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:54:47.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:54:47.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:54:47.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:54:47.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:54:47.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:54:47.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:54:48.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:54:48.029 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:54:48.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:54:48.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:54:48.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:54:48.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T20:54:48.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T20:54:48.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T20:54:48.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:54:48.107 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:54:48.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705666 2026-02-22T20:54:48.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705666 2026-02-22T20:54:48.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-38654705666' 2026-02-22T20:54:48.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:54:48.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T20:54:48.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:54:48.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:54:48.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T20:54:48.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:54:48.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T20:54:48.188 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T20:54:48.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T20:54:48.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:54:48.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T20:54:48.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:54:49.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:54:49.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:54:49.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-22T20:54:49.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:54:49.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705666 2026-02-22T20:54:49.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:54:49.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:54:49.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705666 2026-02-22T20:54:49.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:54:49.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705666 2026-02-22T20:54:49.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705666' 2026-02-22T20:54:49.533 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705666 2026-02-22T20:54:49.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:54:49.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705666 -lt 38654705666 2026-02-22T20:54:49.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:54:49.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:54:49.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:54:49.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:54:49.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:54:49.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:54:49.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:54:49.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:54:49.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:54:49.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:54:49.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:54:50.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:54:50.057 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:54:50.057 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:54:50.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:54:50.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:54:50.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:54:50.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:54:50.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: ceph osd dump 2026-02-22T20:54:50.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: grep '^pool.*['\'']test['\'']' 2026-02-22T20:54:50.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: awk '{ print $2 }' 2026-02-22T20:54:50.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:754: _scrub_snaps_multi: poolid=1 2026-02-22T20:54:50.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:756: _scrub_snaps_multi: dd if=/dev/urandom of=testdata.460661 bs=1032 count=1 2026-02-22T20:54:50.421 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T20:54:50.421 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T20:54:50.421 INFO:tasks.workunit.client.0.trial150.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.4214e-05 s, 30.2 MB/s 2026-02-22T20:54:50.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: seq 1 16 2026-02-22T20:54:50.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj1 testdata.460661 2026-02-22T20:54:50.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj2 testdata.460661 2026-02-22T20:54:50.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj3 testdata.460661 2026-02-22T20:54:50.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj4 testdata.460661 2026-02-22T20:54:50.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj5 testdata.460661 2026-02-22T20:54:50.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj6 testdata.460661 2026-02-22T20:54:50.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj7 testdata.460661 2026-02-22T20:54:50.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj8 testdata.460661 2026-02-22T20:54:50.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj9 testdata.460661 2026-02-22T20:54:50.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj10 testdata.460661 2026-02-22T20:54:50.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj11 testdata.460661 2026-02-22T20:54:50.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj12 testdata.460661 2026-02-22T20:54:50.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj13 testdata.460661 2026-02-22T20:54:50.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj14 testdata.460661 2026-02-22T20:54:50.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj15 testdata.460661 2026-02-22T20:54:50.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:757: _scrub_snaps_multi: for i in `seq 1 $OBJS` 2026-02-22T20:54:50.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:759: _scrub_snaps_multi: rados -p test put obj16 testdata.460661 2026-02-22T20:54:50.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: get_primary test obj1 2026-02-22T20:54:50.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-22T20:54:50.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:54:50.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-22T20:54:50.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:54:50.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:762: _scrub_snaps_multi: local primary=1 2026-02-22T20:54:50.988 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: get_not_primary test obj1 2026-02-22T20:54:50.988 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2026-02-22T20:54:50.988 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-22T20:54:50.989 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2026-02-22T20:54:50.989 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-22T20:54:50.989 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:54:50.989 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-22T20:54:50.989 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:54:51.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T20:54:51.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2026-02-22T20:54:51.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T20:54:51.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:763: _scrub_snaps_multi: local replica=0 2026-02-22T20:54:51.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: eval create_scenario td/osd-scrub-snaps test testdata.460661 '$replica' 2026-02-22T20:54:51.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:765: _scrub_snaps_multi: create_scenario td/osd-scrub-snaps test testdata.460661 0 2026-02-22T20:54:51.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:45: create_scenario: local dir=td/osd-scrub-snaps 2026-02-22T20:54:51.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:46: create_scenario: local poolname=test 2026-02-22T20:54:51.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:47: create_scenario: local TESTDATA=testdata.460661 2026-02-22T20:54:51.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:48: create_scenario: local osd=0 2026-02-22T20:54:51.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:50: create_scenario: SNAP=1 2026-02-22T20:54:51.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:51: create_scenario: rados -p test mksnap snap1 2026-02-22T20:54:51.431 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap1 2026-02-22T20:54:51.434 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:52: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=1 2026-02-22T20:54:51.436 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T20:54:51.436 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T20:54:51.436 INFO:tasks.workunit.client.0.trial150.stderr:256 bytes copied, 8.7534e-05 s, 2.9 MB/s 2026-02-22T20:54:51.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:53: create_scenario: rados -p test put obj1 testdata.460661 2026-02-22T20:54:51.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:54: create_scenario: rados -p test put obj5 testdata.460661 2026-02-22T20:54:51.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:55: create_scenario: rados -p test put obj3 testdata.460661 2026-02-22T20:54:51.516 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: seq 6 14 2026-02-22T20:54:51.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:51.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj6 testdata.460661 2026-02-22T20:54:51.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:51.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj7 testdata.460661 2026-02-22T20:54:51.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:51.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj8 testdata.460661 2026-02-22T20:54:51.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:51.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj9 testdata.460661 2026-02-22T20:54:51.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:51.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj10 testdata.460661 2026-02-22T20:54:51.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:51.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj11 testdata.460661 2026-02-22T20:54:51.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:51.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj12 testdata.460661 2026-02-22T20:54:51.704 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:51.704 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj13 testdata.460661 2026-02-22T20:54:51.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:56: create_scenario: for i in `seq 6 14` 2026-02-22T20:54:51.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:57: create_scenario: rados -p test put obj14 testdata.460661 2026-02-22T20:54:51.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:60: create_scenario: SNAP=2 2026-02-22T20:54:51.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:61: create_scenario: rados -p test mksnap snap2 2026-02-22T20:54:51.785 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap2 2026-02-22T20:54:51.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:62: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=2 2026-02-22T20:54:51.787 INFO:tasks.workunit.client.0.trial150.stderr:2+0 records in 2026-02-22T20:54:51.787 INFO:tasks.workunit.client.0.trial150.stderr:2+0 records out 2026-02-22T20:54:51.787 INFO:tasks.workunit.client.0.trial150.stderr:512 bytes copied, 5.1877e-05 s, 9.9 MB/s 2026-02-22T20:54:51.787 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:63: create_scenario: rados -p test put obj5 testdata.460661 2026-02-22T20:54:51.810 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:65: create_scenario: SNAP=3 2026-02-22T20:54:51.810 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:66: create_scenario: rados -p test mksnap snap3 2026-02-22T20:54:51.886 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap3 2026-02-22T20:54:51.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:67: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=3 2026-02-22T20:54:51.890 INFO:tasks.workunit.client.0.trial150.stderr:3+0 records in 2026-02-22T20:54:51.890 INFO:tasks.workunit.client.0.trial150.stderr:3+0 records out 2026-02-22T20:54:51.890 INFO:tasks.workunit.client.0.trial150.stderr:768 bytes copied, 8.8506e-05 s, 8.7 MB/s 2026-02-22T20:54:51.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:68: create_scenario: rados -p test put obj3 testdata.460661 2026-02-22T20:54:51.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:70: create_scenario: SNAP=4 2026-02-22T20:54:51.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:71: create_scenario: rados -p test mksnap snap4 2026-02-22T20:54:51.989 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap4 2026-02-22T20:54:51.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:72: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=4 2026-02-22T20:54:51.992 INFO:tasks.workunit.client.0.trial150.stderr:4+0 records in 2026-02-22T20:54:51.992 INFO:tasks.workunit.client.0.trial150.stderr:4+0 records out 2026-02-22T20:54:51.992 INFO:tasks.workunit.client.0.trial150.stderr:1024 bytes (1.0 kB, 1.0 KiB) copied, 9.1061e-05 s, 11.2 MB/s 2026-02-22T20:54:51.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:73: create_scenario: rados -p test put obj5 testdata.460661 2026-02-22T20:54:52.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:74: create_scenario: rados -p test put obj2 testdata.460661 2026-02-22T20:54:52.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:76: create_scenario: SNAP=5 2026-02-22T20:54:52.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:77: create_scenario: rados -p test mksnap snap5 2026-02-22T20:54:52.090 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap5 2026-02-22T20:54:52.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:78: create_scenario: SNAP=6 2026-02-22T20:54:52.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:79: create_scenario: rados -p test mksnap snap6 2026-02-22T20:54:52.193 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap6 2026-02-22T20:54:52.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:80: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=6 2026-02-22T20:54:52.196 INFO:tasks.workunit.client.0.trial150.stderr:6+0 records in 2026-02-22T20:54:52.196 INFO:tasks.workunit.client.0.trial150.stderr:6+0 records out 2026-02-22T20:54:52.196 INFO:tasks.workunit.client.0.trial150.stderr:1536 bytes (1.5 kB, 1.5 KiB) copied, 0.000113884 s, 13.5 MB/s 2026-02-22T20:54:52.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:81: create_scenario: rados -p test put obj5 testdata.460661 2026-02-22T20:54:52.224 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:83: create_scenario: SNAP=7 2026-02-22T20:54:52.224 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:84: create_scenario: rados -p test mksnap snap7 2026-02-22T20:54:52.295 INFO:tasks.workunit.client.0.trial150.stdout:created pool test snap snap7 2026-02-22T20:54:52.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:86: create_scenario: rados -p test rm obj4 2026-02-22T20:54:52.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:87: create_scenario: rados -p test rm obj16 2026-02-22T20:54:52.348 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:88: create_scenario: rados -p test rm obj2 2026-02-22T20:54:52.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:90: create_scenario: kill_daemons td/osd-scrub-snaps TERM osd 2026-02-22T20:54:52.375 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:54:52.375 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:54:52.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:54:52.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:54:52.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:54:52.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:54:52.685 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj1 2026-02-22T20:54:52.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:94: create_scenario: JSON='["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:52.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:95: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj1","key":"","snapid":-2,"hash":1828249343,"max":0,"pool":1,"namespace":"","max":0}]' --force remove 2026-02-22T20:54:53.053 INFO:tasks.workunit.client.0.trial150.stdout:WARNING: only removing head with clones present 2026-02-22T20:54:53.053 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:ff7b1f36:::obj1:head# 2026-02-22T20:54:53.169 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-22T20:54:53.169 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: grep '"snapid":2' 2026-02-22T20:54:53.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:97: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:53.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:98: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":2,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-22T20:54:53.506 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:c52c9666:::obj5:2# 2026-02-22T20:54:53.587 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-22T20:54:53.587 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: grep '"snapid":1' 2026-02-22T20:54:53.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:100: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:53.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:101: create_scenario: OBJ5SAVE='["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:53.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:103: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-22T20:54:53.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:104: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:53.973 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:53.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:105: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-22T20:54:53.975 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:54:53.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:106: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype nosnapmap '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-22T20:54:54.171 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:c52c9666:::obj5:1# 2026-02-22T20:54:54.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:108: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-22T20:54:54.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:109: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:54.489 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.0779578A.7.obj4.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:54.490 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:54.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:110: create_scenario: grep '^[pm].*SNA_.*[.]1[.]obj5[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-22T20:54:54.491 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:54:54.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:111: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-22T20:54:54.491 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj5 2026-02-22T20:54:54.491 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: grep '"snapid":4' 2026-02-22T20:54:54.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:113: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:54.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:114: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=18 2026-02-22T20:54:54.669 INFO:tasks.workunit.client.0.trial150.stderr:18+0 records in 2026-02-22T20:54:54.670 INFO:tasks.workunit.client.0.trial150.stderr:18+0 records out 2026-02-22T20:54:54.670 INFO:tasks.workunit.client.0.trial150.stderr:4608 bytes (4.6 kB, 4.5 KiB) copied, 0.00016028 s, 28.7 MB/s 2026-02-22T20:54:54.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:115: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":4,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.460661 2026-02-22T20:54:54.941 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj3 2026-02-22T20:54:55.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:117: create_scenario: JSON='["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:55.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:118: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=15 2026-02-22T20:54:55.121 INFO:tasks.workunit.client.0.trial150.stderr:15+0 records in 2026-02-22T20:54:55.121 INFO:tasks.workunit.client.0.trial150.stderr:15+0 records out 2026-02-22T20:54:55.121 INFO:tasks.workunit.client.0.trial150.stderr:3840 bytes (3.8 kB, 3.8 KiB) copied, 0.000108453 s, 35.4 MB/s 2026-02-22T20:54:55.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:119: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj3","key":"","snapid":-2,"hash":1643547569,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.460661 2026-02-22T20:54:55.389 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj4 2026-02-22T20:54:55.389 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: grep '"snapid":7' 2026-02-22T20:54:55.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:121: create_scenario: JSON='["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:55.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:122: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj4","key":"","snapid":7,"hash":2826278768,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-22T20:54:55.747 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:0ee9ae15:::obj4:7# 2026-02-22T20:54:55.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:125: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-22T20:54:56.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:126: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.049 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:56.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:127: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-22T20:54:56.050 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.050 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.050 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.050 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.050 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.050 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.050 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.268F1DA7.7.obj16.. 2026-02-22T20:54:56.050 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --op list obj16 2026-02-22T20:54:56.051 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: grep '"snapid":7' 2026-02-22T20:54:56.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:128: create_scenario: JSON='["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:56.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:129: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --rmtype snapmap '["1.0",{"oid":"obj16","key":"","snapid":7,"hash":2060580962,"max":0,"pool":1,"namespace":"","max":0}]' remove 2026-02-22T20:54:56.378 INFO:tasks.workunit.client.0.trial150.stdout:remove #1:461f8b5e:::obj16:7# 2026-02-22T20:54:56.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:131: create_scenario: ceph-kvstore-tool bluestore-kv td/osd-scrub-snaps/0 list 2026-02-22T20:54:56.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:132: create_scenario: grep SNA_ td/osd-scrub-snaps/drk.log 2026-02-22T20:54:56.689 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.1BB86F16.1.obj3.. 2026-02-22T20:54:56.689 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.3A439666.1.obj5.. 2026-02-22T20:54:56.689 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.4CC52438.1.obj6.. 2026-02-22T20:54:56.689 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5685B6C7.1.obj13.. 2026-02-22T20:54:56.689 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5AAE7FD8.1.obj8.. 2026-02-22T20:54:56.690 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.5FAF9A3D.1.obj12.. 2026-02-22T20:54:56.690 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.70522149.1.obj14.. 2026-02-22T20:54:56.690 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.77574592.1.obj11.. 2026-02-22T20:54:56.690 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:56.690 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.950988C5.1.obj7.. 2026-02-22T20:54:56.690 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.B8CCECA2.1.obj10.. 2026-02-22T20:54:56.690 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.F7CA874E.1.obj9.. 2026-02-22T20:54:56.690 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000001_0000000000000001.FFED8FC6.1.obj1.. 2026-02-22T20:54:56.690 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:56.690 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000002_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:56.690 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.1BB86F16.3.obj3.. 2026-02-22T20:54:56.690 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:56.691 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000003_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:56.691 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.3A439666.4.obj5.. 2026-02-22T20:54:56.691 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000004_0000000000000001.802EE1F3.4.obj2.. 2026-02-22T20:54:56.691 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:56.691 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000005_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:56.691 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.3A439666.6.obj5.. 2026-02-22T20:54:56.691 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000006_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:56.691 INFO:tasks.workunit.client.0.trial150.stdout:p %00%00%00%00%00%00%00%00%c07%16%25%00%00%00%00%00%00%04%02.SNA_1_0000000000000007_0000000000000001.802EE1F3.7.obj2.. 2026-02-22T20:54:56.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:133: create_scenario: grep '^[pm].*SNA_.*[.]7[.]obj16[.][.]$' td/osd-scrub-snaps/drk.log 2026-02-22T20:54:56.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:134: create_scenario: rm -f td/osd-scrub-snaps/drk.log 2026-02-22T20:54:56.692 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj2 2026-02-22T20:54:56.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:136: create_scenario: JSON='["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:56.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:137: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj2","key":"","snapid":-2,"hash":1058988552,"max":0,"pool":1,"namespace":"","max":0}]' rm-attr snapset 2026-02-22T20:54:57.121 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: echo '["1.0",{"oid":"obj5","key":"","snapid":1,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:57.122 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: sed 's/snapid":1/snapid":7/' 2026-02-22T20:54:57.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:140: create_scenario: JSON='["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:57.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:141: create_scenario: dd if=/dev/urandom of=testdata.460661 bs=256 count=7 2026-02-22T20:54:57.124 INFO:tasks.workunit.client.0.trial150.stderr:7+0 records in 2026-02-22T20:54:57.124 INFO:tasks.workunit.client.0.trial150.stderr:7+0 records out 2026-02-22T20:54:57.124 INFO:tasks.workunit.client.0.trial150.stderr:1792 bytes (1.8 kB, 1.8 KiB) copied, 0.000102823 s, 17.4 MB/s 2026-02-22T20:54:57.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:142: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj5","key":"","snapid":7,"hash":1718170787,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes testdata.460661 2026-02-22T20:54:57.413 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj6 2026-02-22T20:54:57.507 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:57.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:144: create_scenario: JSON='["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:57.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:145: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj6","key":"","snapid":-2,"hash":2202164420,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset 2026-02-22T20:54:57.901 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj7 2026-02-22T20:54:57.987 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:58.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:146: create_scenario: JSON='["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:58.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:147: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj7","key":"","snapid":-2,"hash":1552453721,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset corrupt 2026-02-22T20:54:58.333 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj8 2026-02-22T20:54:58.411 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:58.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:148: create_scenario: JSON='["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:58.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:149: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj8","key":"","snapid":-2,"hash":2381834917,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset seq 2026-02-22T20:54:58.752 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj9 2026-02-22T20:54:58.816 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:58.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:150: create_scenario: JSON='["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:58.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:151: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj9","key":"","snapid":-2,"hash":3833113727,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_size 2026-02-22T20:54:59.225 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj10 2026-02-22T20:54:59.311 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:59.412 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:152: create_scenario: JSON='["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:59.412 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:153: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj10","key":"","snapid":-2,"hash":718195851,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clone_overlap 2026-02-22T20:54:59.701 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj11 2026-02-22T20:54:59.780 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:54:59.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:154: create_scenario: JSON='["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:54:59.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:155: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj11","key":"","snapid":-2,"hash":693400951,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset clones 2026-02-22T20:55:00.155 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj12 2026-02-22T20:55:00.221 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:55:00.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:156: create_scenario: JSON='["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:55:00.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:157: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj12","key":"","snapid":-2,"hash":3551132405,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset head 2026-02-22T20:55:00.565 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj13 2026-02-22T20:55:00.631 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:55:00.720 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:158: create_scenario: JSON='["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:55:00.720 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:159: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj13","key":"","snapid":-2,"hash":2087409765,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset snaps 2026-02-22T20:55:00.993 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj14 2026-02-22T20:55:01.064 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:55:01.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:160: create_scenario: JSON='["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:55:01.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:161: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj14","key":"","snapid":-2,"hash":2484217095,"max":0,"pool":1,"namespace":"","max":0}]' clear-snapset size 2026-02-22T20:55:01.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:163: create_scenario: echo garbage 2026-02-22T20:55:01.453 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 --head --op list obj15 2026-02-22T20:55:01.531 INFO:tasks.workunit.client.0.trial150.stderr:Error getting attr on : 1.0_head,#1:c52c9666:::obj5:7#, (61) No data available 2026-02-22T20:55:01.632 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:164: create_scenario: JSON='["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T20:55:01.632 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:165: create_scenario: ceph-objectstore-tool --data-path td/osd-scrub-snaps/0 '["1.0",{"oid":"obj15","key":"","snapid":-2,"hash":612772309,"max":0,"pool":1,"namespace":"","max":0}]' set-attr snapset td/osd-scrub-snaps/bad 2026-02-22T20:55:01.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:166: create_scenario: rm -f td/osd-scrub-snaps/bad 2026-02-22T20:55:01.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:167: create_scenario: return 0 2026-02-22T20:55:01.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:767: _scrub_snaps_multi: rm -f testdata.460661 2026-02-22T20:55:01.899 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: expr 2 - 1 2026-02-22T20:55:01.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: seq 0 1 2026-02-22T20:55:01.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:55:01.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 0 2026-02-22T20:55:01.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-22T20:55:01.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:55:01.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-22T20:55:01.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:55:01.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/0 2026-02-22T20:55:01.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-22T20:55:01.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:55:01.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:55:01.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:55:01.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/0' 2026-02-22T20:55:01.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/0/journal' 2026-02-22T20:55:01.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:55:01.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:55:01.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-22T20:55:01.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:55:01.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:55:01.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:55:01.905 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:55:01.905 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:55:01.905 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:55:01.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:55:01.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:55:01.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:55:01.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-22T20:55:01.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-22T20:55:01.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:55:01.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:55:01.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:55:01.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:55:01.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:55:01.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:55:01.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/0 2026-02-22T20:55:01.908 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:55:01.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-22T20:55:01.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/0 --osd-journal=td/osd-scrub-snaps/0/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:55:01.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/0/whoami 2026-02-22T20:55:01.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-22T20:55:01.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:55:01.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:55:01.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:55:01.931 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:01.921+0000 7f3a31e3ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:01.932 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:01.921+0000 7f3a31e3ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:01.933 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:01.925+0000 7f3a31e3ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:02.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-22T20:55:02.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:55:02.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:55:02.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:55:02.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:55:02.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:55:02.071 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:55:02.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:55:02.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:55:02.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:55:02.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:55:02.290 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:02.281+0000 7f3a31e3ea40 -1 Falling back to public interface 2026-02-22T20:55:02.952 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:02.941+0000 7f3a31e3ea40 -1 osd.0 21 log_to_monitors true 2026-02-22T20:55:03.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:55:03.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:55:03.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:55:03.236 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:55:03.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:55:03.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:55:03.399 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 24 up_thru 24 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6802/3819938602,v1:127.0.0.1:6803/3819938602] [v2:127.0.0.1:6804/3819938602,v1:127.0.0.1:6805/3819938602] exists,up 2efe00df-e546-4390-961e-5210e1a63256 2026-02-22T20:55:03.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:55:03.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:55:03.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:55:03.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:769: _scrub_snaps_multi: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:55:03.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:771: _scrub_snaps_multi: activate_osd td/osd-scrub-snaps 1 2026-02-22T20:55:03.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-snaps 2026-02-22T20:55:03.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T20:55:03.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T20:55:03.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T20:55:03.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-snaps/1 2026-02-22T20:55:03.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1' 2026-02-22T20:55:03.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:55:03.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:55:03.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:55:03.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-snaps/1' 2026-02-22T20:55:03.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-snaps/1/journal' 2026-02-22T20:55:03.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T20:55:03.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T20:55:03.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-snaps' 2026-02-22T20:55:03.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T20:55:03.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:55:03.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:55:03.405 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:55:03.405 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:55:03.405 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:55:03.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:55:03.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' 2026-02-22T20:55:03.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:55:03.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-snaps/$name.log' 2026-02-22T20:55:03.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-snaps/$name.pid' 2026-02-22T20:55:03.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:55:03.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:55:03.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:55:03.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:55:03.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T20:55:03.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T20:55:03.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-snaps/1 2026-02-22T20:55:03.406 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:55:03.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T20:55:03.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 --osd_scrub_chunk_min=3 --osd_scrub_chunk_max=20 --osd_shallow_scrub_chunk_min=3 --osd_shallow_scrub_chunk_max=3 --osd_pg_stat_report_interval_max_seconds=1 --osd_pg_stat_report_interval_max_epochs=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-snaps/1 --osd-journal=td/osd-scrub-snaps/1/journal --chdir= --run-dir=td/osd-scrub-snaps '--admin-socket=/tmp/ceph-asok.460661/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-snaps/$name.log' '--pid-file=td/osd-scrub-snaps/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T20:55:03.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-snaps/1/whoami 2026-02-22T20:55:03.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T20:55:03.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T20:55:03.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T20:55:03.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T20:55:03.424 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:03.413+0000 7f6b9edf3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:03.425 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:03.417+0000 7f6b9edf3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:03.426 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:03.417+0000 7f6b9edf3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:03.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T20:55:03.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:55:03.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:55:03.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:55:03.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:55:03.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:55:03.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:55:03.576 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:55:03.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:55:03.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:55:03.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:55:03.830 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:03.821+0000 7f6b9edf3a40 -1 Falling back to public interface 2026-02-22T20:55:04.479 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:04.469+0000 7f6b9edf3a40 -1 osd.1 21 log_to_monitors true 2026-02-22T20:55:04.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:55:04.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:55:04.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:55:04.737 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:55:04.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:55:04.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:55:04.900 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 28 up_thru 28 down_at 22 last_clean_interval [9,21) [v2:127.0.0.1:6810/396925782,v1:127.0.0.1:6811/396925782] [v2:127.0.0.1:6812/396925782,v1:127.0.0.1:6813/396925782] exists,up 56b865e5-5595-402d-af0e-2f659bcb2e2c 2026-02-22T20:55:04.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:55:04.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:55:04.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:55:04.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:774: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2026-02-22T20:55:04.972 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:04.972 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:55:04.973 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:04.980 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:04.980 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_shallow_scrub_chunk_max = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-22T20:55:04.980 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:04.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:775: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_min 3 2026-02-22T20:55:05.071 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:05.072 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-22T20:55:05.072 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:05.079 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:05.079 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_min = '' " 2026-02-22T20:55:05.079 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:05.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:776: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_scrub_chunk_min 3 2026-02-22T20:55:05.166 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:05.166 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_min = '' " 2026-02-22T20:55:05.166 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:05.174 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:05.174 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_min = '' " 2026-02-22T20:55:05.174 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:05.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:777: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_seconds 1 2026-02-22T20:55:05.254 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:05.254 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-22T20:55:05.254 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:05.261 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:05.261 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_pg_stat_report_interval_max_seconds = '' " 2026-02-22T20:55:05.261 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:05.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:778: _scrub_snaps_multi: ceph tell 'osd.*' config set osd_pg_stat_report_interval_max_epochs 1 2026-02-22T20:55:05.348 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:05.348 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-22T20:55:05.348 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:05.355 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:05.355 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_pg_stat_report_interval_max_epochs = '' " 2026-02-22T20:55:05.355 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:05.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:779: _scrub_snaps_multi: wait_for_clean 2026-02-22T20:55:05.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:55:05.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:55:05.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:55:05.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:55:05.364 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:55:05.364 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:55:05.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:55:05.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:55:05.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:55:05.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:55:05.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:55:05.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:55:05.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:55:05.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:55:05.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:55:05.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:55:05.647 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:55:05.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:55:05.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:55:05.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:55:05.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215106 2026-02-22T20:55:05.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215106 2026-02-22T20:55:05.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106' 2026-02-22T20:55:05.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:55:05.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:55:05.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084290 2026-02-22T20:55:05.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084290 2026-02-22T20:55:05.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215106 1-120259084290' 2026-02-22T20:55:05.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:55:05.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215106 2026-02-22T20:55:05.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:55:05.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:55:05.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215106 2026-02-22T20:55:05.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:55:05.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215106 2026-02-22T20:55:05.804 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 103079215106 2026-02-22T20:55:05.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215106' 2026-02-22T20:55:05.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:55:05.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-22T20:55:05.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:55:06.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:55:06.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:55:07.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 103079215106 2026-02-22T20:55:07.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:55:08.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:55:08.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:55:08.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215106 -lt 103079215106 2026-02-22T20:55:08.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:55:08.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084290 2026-02-22T20:55:08.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:55:08.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:55:08.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084290 2026-02-22T20:55:08.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:55:08.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084290 2026-02-22T20:55:08.313 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 120259084290 2026-02-22T20:55:08.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084290' 2026-02-22T20:55:08.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:55:08.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084290 -lt 120259084290 2026-02-22T20:55:08.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:55:08.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:55:08.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:55:08.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:55:08.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:55:08.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:55:08.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:55:08.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:55:08.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:55:08.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:55:08.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:55:08.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:55:08.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:55:08.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:55:08.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:55:09.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:55:09.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:55:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:55:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:781: _scrub_snaps_multi: local pgid=1.0 2026-02-22T20:55:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:782: _scrub_snaps_multi: pg_scrub 1.0 2026-02-22T20:55:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-22T20:55:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-22T20:55:09.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-22T20:55:09.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T20:55:09.045 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:55:09.045 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:55:09.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:55:09.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:55:09.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:55:09.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T20:55:09.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T20:55:09.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T20:55:09.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T20:55:09.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:55:09.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:55:09.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:55:09.478 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T20:55:09.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:55:09.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:55:09.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:55:09.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=103079215108 2026-02-22T20:55:09.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 103079215108 2026-02-22T20:55:09.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108' 2026-02-22T20:55:09.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:55:09.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:55:09.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084292 2026-02-22T20:55:09.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084292 2026-02-22T20:55:09.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-103079215108 1-120259084292' 2026-02-22T20:55:09.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:55:09.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-103079215108 2026-02-22T20:55:09.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:55:09.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:55:09.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-103079215108 2026-02-22T20:55:09.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:55:09.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=103079215108 2026-02-22T20:55:09.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 103079215108' 2026-02-22T20:55:09.636 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 103079215108 2026-02-22T20:55:09.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:55:09.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215107 -lt 103079215108 2026-02-22T20:55:09.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:55:10.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:55:10.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:55:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215107 -lt 103079215108 2026-02-22T20:55:10.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:55:11.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:55:11.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:55:12.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 103079215108 -lt 103079215108 2026-02-22T20:55:12.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:55:12.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084292 2026-02-22T20:55:12.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:55:12.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:55:12.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084292 2026-02-22T20:55:12.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:55:12.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084292 2026-02-22T20:55:12.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084292' 2026-02-22T20:55:12.135 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 120259084292 2026-02-22T20:55:12.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:55:12.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084292 -lt 120259084292 2026-02-22T20:55:12.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T20:55:12.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-22T20:55:12.309 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 1.0 loop 0 2026-02-22T20:55:12.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T20:55:12.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-22T20:55:12.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T20:55:12.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-22T20:55:12.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T20:55:12.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T20:55:12.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T20:55:12.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T20:55:12.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T20:55:12.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-22T20:55:12.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:55:12.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:55:12.391 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:55:12.391 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:55:12.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-22T20:54:46.733739+0000 2026-02-22T20:55:12.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-22T20:55:12.707 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0 on osd.1 to scrub 2026-02-22T20:55:12.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-22T20:54:46.733739+0000 2026-02-22T20:55:12.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:55:12.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:54:46.733739+0000 2026-02-22T20:55:12.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:55:12.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:55:12.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:55:12.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:55:12.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:55:12.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:55:12.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:55:12.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:55:12.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:54:46.733739+0000 '>' 2026-02-22T20:54:46.733739+0000 2026-02-22T20:55:12.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:55:13.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:55:13.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:55:13.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:55:13.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:55:13.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:55:13.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:55:13.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:55:14.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:54:46.733739+0000 '>' 2026-02-22T20:54:46.733739+0000 2026-02-22T20:55:14.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:55:15.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:55:15.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-22T20:55:15.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:55:15.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:55:15.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:55:15.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:55:15.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:55:15.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:55:13.490610+0000 '>' 2026-02-22T20:54:46.733739+0000 2026-02-22T20:55:15.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T20:55:15.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:55:15.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: wc -l 2026-02-22T20:55:15.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:786: _scrub_snaps_multi: test 16 -gt 3 2026-02-22T20:55:15.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: grep '_scan_snaps start' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:55:15.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: wc -l 2026-02-22T20:55:15.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:787: _scrub_snaps_multi: test 16 -gt 3 2026-02-22T20:55:15.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:789: _scrub_snaps_multi: rados list-inconsistent-pg test 2026-02-22T20:55:15.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: jq '. | length' td/osd-scrub-snaps/json 2026-02-22T20:55:15.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:791: _scrub_snaps_multi: test 1 = 1 2026-02-22T20:55:15.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: jq -r '.[0]' td/osd-scrub-snaps/json 2026-02-22T20:55:15.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:793: _scrub_snaps_multi: test 1.0 = 1.0 2026-02-22T20:55:15.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:795: _scrub_snaps_multi: rados list-inconsistent-obj 1.0 --format=json-pretty 2026-02-22T20:55:15.288 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:15.288 INFO:tasks.workunit.client.0.trial150.stdout: "epoch": 28, 2026-02-22T20:55:15.288 INFO:tasks.workunit.client.0.trial150.stdout: "inconsistents": [ 2026-02-22T20:55:15.288 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.288 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:55:15.288 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj1", 2026-02-22T20:55:15.288 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:55:15.288 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:55:15.288 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:55:15.288 INFO:tasks.workunit.client.0.trial150.stdout: "version": 18 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [ 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "missing" 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj1", 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 1828249343, 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'18", 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'1", 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4222.0:1", 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 18, 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:51.449663+0000", 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:51.450912+0000", 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.289 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xa01a38a3", 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "missing" 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.290 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj10", 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "version": 32 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj10", 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 718195851, 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'32", 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'10", 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4243.0:1", 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 32, 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:51.646251+0000", 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:51.646685+0000", 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.291 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xa01a38a3", 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "????", 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.292 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj11", 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "version": 34 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:55:15.293 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj11", 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 693400951, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'34", 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'11", 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4246.0:1", 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 34, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:51.674741+0000", 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:51.675178+0000", 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xa01a38a3", 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.294 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [] 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.295 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj13", 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "version": 38 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj13", 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 2087409765, 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'38", 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'13", 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4252.0:1", 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 38, 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:51.725950+0000", 2026-02-22T20:55:15.296 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:51.726350+0000", 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xa01a38a3", 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]" 2026-02-22T20:55:15.297 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj14", 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "version": 40 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.298 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj14", 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 2484217095, 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'40", 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'14", 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4255.0:1", 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 40, 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:51.753120+0000", 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:51.753542+0000", 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xa01a38a3", 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.299 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1033, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.300 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj15", 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "version": 15 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [ 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_corrupted" 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj15", 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.301 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 612772309, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "version": "14'15", 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "0'0", 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4207.0:1", 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 15, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:50.788192+0000", 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:50.788576+0000", 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xc8de8208", 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:55:15.302 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_corrupted" 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": "Z2FyYmFnZQo=" 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 0, 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [] 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj2", 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "version": 56 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [ 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_missing" 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj2", 2026-02-22T20:55:15.303 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 1058988552, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "version": "21'56", 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "18'48", 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4297.0:1", 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 56, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "size": 0, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:52.367760+0000", 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:52.368089+0000", 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "whiteout", 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "dirty" 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xffffffff", 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:55:15.304 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_missing" 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "size": 0 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "size": 0, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 7, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 4, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: 4, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: 3, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: 2, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 7, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1024, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: 7, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: 6, 2026-02-22T20:55:15.305 INFO:tasks.workunit.client.0.trial150.stdout: 5 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj3", 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "version": 44 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "size_mismatch" 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [ 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "size_mismatch_info", 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "obj_size_info_mismatch" 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj3", 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 1643547569, 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.306 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "version": "17'44", 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "15'22", 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4267.0:1", 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 44, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "size": 768, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:51.911275+0000", 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:51.912366+0000", 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xfecbccec", 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "size_mismatch_info", 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "obj_size_info_mismatch" 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.307 INFO:tasks.workunit.client.0.trial150.stdout: "size": 3840, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "object_info": { 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj3", 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 1643547569, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "version": "17'44", 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "15'22", 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4267.0:1", 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 44, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "size": 768, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:51.911275+0000", 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:51.912366+0000", 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xfecbccec", 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.308 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "size": 768 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj6", 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "version": 24 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj6", 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 2202164420, 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.309 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'24", 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'6", 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4231.0:1", 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 24, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:51.532337+0000", 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:51.532647+0000", 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xa01a38a3", 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.310 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [] 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj7", 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:55:15.311 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "version": 26 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj7", 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 1552453721, 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'26", 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'7", 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4234.0:1", 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 26, 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:51.562013+0000", 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:51.562403+0000", 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.312 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xa01a38a3", 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 0, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [] 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.313 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj8", 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "version": 28 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj8", 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 2381834917, 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.314 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'28", 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'8", 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4237.0:1", 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 28, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:51.589203+0000", 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:51.589636+0000", 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xa01a38a3", 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.315 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.316 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.316 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 0, 2026-02-22T20:55:15.316 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.316 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.316 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.316 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.316 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.316 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.316 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.316 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.316 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.317 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "object": { 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "name": "obj9", 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "nspace": "", 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "locator": "", 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "snap": "head", 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "version": 30 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [ 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "snapset_inconsistency" 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "union_shard_errors": [], 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "selected_object_info": { 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "oid": { 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "oid": "obj9", 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "key": "", 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "snapid": -2, 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "hash": 3833113727, 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "max": 0, 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "pool": 1, 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "namespace": "" 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "version": "15'30", 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "prior_version": "14'9", 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "last_reqid": "client.4240.0:1", 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "user_version": 30, 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "mtime": "2026-02-22T20:54:51.617654+0000", 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "local_mtime": "2026-02-22T20:54:51.618042+0000", 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "lost": 0, 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "flags": [ 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "dirty", 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest" 2026-02-22T20:55:15.318 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_seq": 0, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "truncate_size": 0, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "data_digest": "0xa01a38a3", 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "omap_digest": "0xffffffff", 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "expected_object_size": 0, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "expected_write_size": 0, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "alloc_hint_flags": 0, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "manifest": { 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "type": 0 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "watchers": {}, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "shard_versions": [] 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "shards": [ 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 0, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "primary": false, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "size": "????", 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.319 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: "osd": 1, 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: "primary": true, 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: "errors": [], 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: "size": 256, 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: "snapset": { 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: "seq": 1, 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: "clones": [ 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: "snap": 1, 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: "size": 1032, 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: "overlap": "[]", 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: "snaps": [ 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: 1 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:55:15.320 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:15.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:797: _scrub_snaps_multi: rados list-inconsistent-snapset 1.0 2026-02-22T20:55:15.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:800: _scrub_snaps_multi: '[' replica = replica ']' 2026-02-22T20:55:15.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:802: _scrub_snaps_multi: scruberrors=20 2026-02-22T20:55:15.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: jq .inconsistents 2026-02-22T20:55:15.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:803: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-22T20:55:15.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: jq .inconsistents td/osd-scrub-snaps/json 2026-02-22T20:55:15.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1066: _scrub_snaps_multi: python3 -c 'import json; import sys ; JSON=sys.stdin.read() ; ud = json.loads(JSON) ; print ( json.dumps(ud, sort_keys=True, indent=2) )' 2026-02-22T20:55:15.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1067: _scrub_snaps_multi: multidiff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-22T20:55:15.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2513: multidiff: diff td/osd-scrub-snaps/checkcsjson td/osd-scrub-snaps/csjson 2026-02-22T20:55:15.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1068: _scrub_snaps_multi: test no = yes 2026-02-22T20:55:15.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1073: _scrub_snaps_multi: test '' = yes 2026-02-22T20:55:15.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: find td/osd-scrub-snaps 2026-02-22T20:55:15.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: grep 'osd[^/]*\.pid' 2026-02-22T20:55:15.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1078: _scrub_snaps_multi: pidfiles='td/osd-scrub-snaps/osd.1.pid 2026-02-22T20:55:15.347 INFO:tasks.workunit.client.0.trial150.stderr:td/osd-scrub-snaps/osd.0.pid' 2026-02-22T20:55:15.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1079: _scrub_snaps_multi: pids= 2026-02-22T20:55:15.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2026-02-22T20:55:15.348 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.1.pid 2026-02-22T20:55:15.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='514832 ' 2026-02-22T20:55:15.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1080: _scrub_snaps_multi: for pidfile in ${pidfiles} 2026-02-22T20:55:15.349 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: cat td/osd-scrub-snaps/osd.0.pid 2026-02-22T20:55:15.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1082: _scrub_snaps_multi: pids+='514290 ' 2026-02-22T20:55:15.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1085: _scrub_snaps_multi: ERRORS=0 2026-02-22T20:55:15.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1089: _scrub_snaps_multi: '[' replica = primary ']' 2026-02-22T20:55:15.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1111: _scrub_snaps_multi: ceph pg dump pgs 2026-02-22T20:55:15.499 INFO:tasks.workunit.client.0.trial150.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:55:15.499 INFO:tasks.workunit.client.0.trial150.stdout:1.0 36 0 0 0 0 24448 0 0 56 0 56 active+clean+inconsistent 2026-02-22T20:55:13.490671+0000 21'56 30:107 [1,0] 1 [1,0] 1 21'56 2026-02-22T20:55:13.490610+0000 0'0 2026-02-22T20:54:46.733739+0000 0 1 periodic scrub scheduled @ 2026-02-24T02:25:30.354968+0000 36 0 2026-02-22T20:55:15.499 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:55:15.499 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:55:15.499 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:15.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2026-02-22T20:55:15.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 514832 2026-02-22T20:55:15.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1113: _scrub_snaps_multi: for pid in $pids 2026-02-22T20:55:15.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1115: _scrub_snaps_multi: kill -0 514290 2026-02-22T20:55:15.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1122: _scrub_snaps_multi: kill_daemons td/osd-scrub-snaps 2026-02-22T20:55:15.508 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:55:15.508 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:55:15.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:55:15.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:55:15.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:55:20.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:55:20.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1124: _scrub_snaps_multi: declare -a err_strings 2026-02-22T20:55:20.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1125: _scrub_snaps_multi: err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' 2026-02-22T20:55:20.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1126: _scrub_snaps_multi: err_strings[1]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' 2026-02-22T20:55:20.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1127: _scrub_snaps_multi: err_strings[2]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' 2026-02-22T20:55:20.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1128: _scrub_snaps_multi: err_strings[3]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' 2026-02-22T20:55:20.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1129: _scrub_snaps_multi: err_strings[4]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' 2026-02-22T20:55:20.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1130: _scrub_snaps_multi: err_strings[5]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' 2026-02-22T20:55:20.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1131: _scrub_snaps_multi: err_strings[6]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' 2026-02-22T20:55:20.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1132: _scrub_snaps_multi: err_strings[7]='log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' 2026-02-22T20:55:20.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:55:20.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj4:7 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:55:20.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:55:20.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj3:head : size 3840 != size 768 from auth oi' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:55:20.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:55:20.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:1 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:55:20.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:55:20.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj5:2 : missing' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:55:20.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:55:20.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] soid .*:::obj5:4 : size 4608 != size 512 from auth oi' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:55:20.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:55:20.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 soid .*:::obj5:7 : failed to pick suitable object info' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:55:20.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:55:20.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 shard [0-1] .*:::obj1:head : missing' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:55:20.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1134: _scrub_snaps_multi: for err_string in "${err_strings[@]}" 2026-02-22T20:55:20.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1136: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : [0-9]*[.]0 scrub 20 errors' td/osd-scrub-snaps/osd.1.log 2026-02-22T20:55:20.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1144: _scrub_snaps_multi: declare -a rep_err_strings 2026-02-22T20:55:20.845 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: eval echo '$replica' 2026-02-22T20:55:20.845 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: echo 0 2026-02-22T20:55:20.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1145: _scrub_snaps_multi: osd=0 2026-02-22T20:55:20.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1146: _scrub_snaps_multi: rep_err_strings[0]='log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' 2026-02-22T20:55:20.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1147: _scrub_snaps_multi: for err_string in "${rep_err_strings[@]}" 2026-02-22T20:55:20.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1149: _scrub_snaps_multi: grep 'log_channel[(]cluster[)] log [[]ERR[]] : osd[.][0-9]* found snap mapper error on pg 1.0 oid 1:461f8b5e:::obj16:7 snaps missing in mapper, should be: {1, 2, 3, 4, 5, 6, 7} ...repaired' td/osd-scrub-snaps/osd.0.log 2026-02-22T20:55:20.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1156: _scrub_snaps_multi: '[' 0 '!=' 0 ']' 2026-02-22T20:55:20.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1162: _scrub_snaps_multi: echo 'TEST PASSED' 2026-02-22T20:55:20.849 INFO:tasks.workunit.client.0.trial150.stdout:TEST PASSED 2026-02-22T20:55:20.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1163: _scrub_snaps_multi: return 0 2026-02-22T20:55:20.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1171: TEST_scrub_snaps_replica: err=0 2026-02-22T20:55:20.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1172: TEST_scrub_snaps_replica: CEPH_ARGS='--fsid=05fa9275-d3e5-46ee-a679-6f00fc0f2bdd --auth-supported=none --mon-host=127.0.0.1:7121 ' 2026-02-22T20:55:20.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:1173: TEST_scrub_snaps_replica: return 0 2026-02-22T20:55:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-snaps.sh:40: run: teardown td/osd-scrub-snaps 2026-02-22T20:55:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-22T20:55:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:55:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-22T20:55:20.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:55:20.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:55:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:55:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:55:20.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:55:20.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:55:20.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:55:20.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:55:20.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:55:20.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:55:20.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:55:20.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:55:20.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:55:20.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:55:20.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:55:20.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:55:20.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:55:20.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:55:20.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-22T20:55:20.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:55:20.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:55:20.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:55:20.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.460661 2026-02-22T20:55:20.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:55:20.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:55:20.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-22T20:55:20.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-scrub-snaps 0 2026-02-22T20:55:20.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-snaps 2026-02-22T20:55:20.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-22T20:55:20.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-snaps KILL 2026-02-22T20:55:20.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:55:20.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:55:20.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:55:20.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:55:20.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:55:20.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:55:20.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:55:20.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:55:20.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:55:20.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:55:20.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:55:20.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:55:20.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:55:20.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:55:20.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:55:20.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:55:20.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:55:20.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T20:55:20.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-snaps 2026-02-22T20:55:20.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:55:20.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:55:20.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.460661 2026-02-22T20:55:20.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.460661 2026-02-22T20:55:20.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:55:20.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:55:20.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-22T20:55:20.892 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T20:55:20.892 DEBUG:teuthology.orchestra.run.trial150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T20:55:20.941 INFO:tasks.workunit:Running workunit scrub/osd-scrub-test.sh... 2026-02-22T20:55:20.942 DEBUG:teuthology.orchestra.run.trial150:workunit test scrub/osd-scrub-test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5c3084f688b1f6eff223b8d259a8316f9844c8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh 2026-02-22T20:55:20.994 INFO:tasks.workunit.client.0.trial150.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T20:55:20.999 INFO:tasks.workunit.client.0.trial150.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T20:55:20.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T20:55:20.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T20:55:20.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-22T20:55:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-22T20:55:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-22T20:55:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-22T20:55:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-22T20:55:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-22T20:55:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-scrub-test 2026-02-22T20:55:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:21: run: local dir=td/osd-scrub-test 2026-02-22T20:55:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:22: run: shift 2026-02-22T20:55:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: export CEPH_MON=127.0.0.1:7138 2026-02-22T20:55:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:24: run: CEPH_MON=127.0.0.1:7138 2026-02-22T20:55:21.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:25: run: export CEPH_ARGS 2026-02-22T20:55:21.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: uuidgen 2026-02-22T20:55:21.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:26: run: CEPH_ARGS+='--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none ' 2026-02-22T20:55:21.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7138 ' 2026-02-22T20:55:21.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:29: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-22T20:55:21.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: set 2026-02-22T20:55:21.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T20:55:21.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:30: run: local 'funcs=TEST_abort_periodic_for_operator 2026-02-22T20:55:21.003 INFO:tasks.workunit.client.0.trial150.stderr:TEST_deep_scrub_abort 2026-02-22T20:55:21.003 INFO:tasks.workunit.client.0.trial150.stderr:TEST_dump_scrub_schedule 2026-02-22T20:55:21.003 INFO:tasks.workunit.client.0.trial150.stderr:TEST_interval_changes 2026-02-22T20:55:21.003 INFO:tasks.workunit.client.0.trial150.stderr:TEST_just_deep_scrubs 2026-02-22T20:55:21.003 INFO:tasks.workunit.client.0.trial150.stderr:TEST_pg_dump_objects_scrubbed 2026-02-22T20:55:21.003 INFO:tasks.workunit.client.0.trial150.stderr:TEST_scrub_abort 2026-02-22T20:55:21.003 INFO:tasks.workunit.client.0.trial150.stderr:TEST_scrub_permit_time 2026-02-22T20:55:21.003 INFO:tasks.workunit.client.0.trial150.stderr:TEST_scrub_test' 2026-02-22T20:55:21.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-22T20:55:21.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_abort_periodic_for_operator -------------------' 2026-02-22T20:55:21.004 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Prepare Test TEST_abort_periodic_for_operator ------------------- 2026-02-22T20:55:21.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-22T20:55:21.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-22T20:55:21.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-22T20:55:21.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T20:55:21.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:55:21.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T20:55:21.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:55:21.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:55:21.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:55:21.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:55:21.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:55:21.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:55:21.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:55:21.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:55:21.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:55:21.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:55:21.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:55:21.009 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:55:21.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:55:21.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:55:21.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:55:21.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:55:21.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:55:21.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:55:21.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T20:55:21.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:55:21.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:55:21.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:55:21.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T20:55:21.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:55:21.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:55:21.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-22T20:55:21.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:55:21.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:55:21.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:55:21.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.516403 2026-02-22T20:55:21.018 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:55:21.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-22T20:55:21.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-22T20:55:21.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:55:21.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-22T20:55:21.020 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Run Test TEST_abort_periodic_for_operator ----------------------- 2026-02-22T20:55:21.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_abort_periodic_for_operator -----------------------' 2026-02-22T20:55:21.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_abort_periodic_for_operator td/osd-scrub-test 2026-02-22T20:55:21.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:748: TEST_abort_periodic_for_operator: local dir=td/osd-scrub-test 2026-02-22T20:55:21.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:749: TEST_abort_periodic_for_operator: cluster_conf=(['osds_num']='5' ['pgs_in_pool']='16' ['pool_name']='test') 2026-02-22T20:55:21.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:749: TEST_abort_periodic_for_operator: local -A cluster_conf 2026-02-22T20:55:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:754: TEST_abort_periodic_for_operator: local extr_dbg=1 2026-02-22T20:55:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:756: TEST_abort_periodic_for_operator: standard_scrub_wpq_cluster td/osd-scrub-test cluster_conf 2 2026-02-22T20:55:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:302: standard_scrub_wpq_cluster: local dir=td/osd-scrub-test 2026-02-22T20:55:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:303: standard_scrub_wpq_cluster: local -n conf=cluster_conf 2026-02-22T20:55:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:304: standard_scrub_wpq_cluster: local osd_sleep=2 2026-02-22T20:55:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:306: standard_scrub_wpq_cluster: conf['extras']=' --osd_op_queue=wpq --osd_scrub_sleep=2 ' 2026-02-22T20:55:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:308: standard_scrub_wpq_cluster: standard_scrub_cluster td/osd-scrub-test conf 2026-02-22T20:55:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-test 2026-02-22T20:55:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=conf 2026-02-22T20:55:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=5 2026-02-22T20:55:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=16 2026-02-22T20:55:21.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=test 2026-02-22T20:55:21.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=test 2026-02-22T20:55:21.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=3 2026-02-22T20:55:21.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=3 2026-02-22T20:55:21.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local 'extra_pars= --osd_op_queue=wpq --osd_scrub_sleep=2 ' 2026-02-22T20:55:21.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-22T20:55:21.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-22T20:55:21.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-22T20:55:21.322 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 313477cb-fdcc-42dc-ae1a-2a329442519f 2026-02-22T20:55:21.427 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:55:21.456 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:21.449+0000 7fbe19364a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:21.458 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:21.449+0000 7fbe19364a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:21.459 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:21.449+0000 7fbe19364a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:21.459 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:21.449+0000 7fbe19364a40 -1 bdev(0x559e0e5f4c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:55:21.459 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:21.449+0000 7fbe19364a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-22T20:55:22.264 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:55:22.468 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:55:22.501 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:22.493+0000 7fd5d7925a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:22.502 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:22.493+0000 7fd5d7925a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:22.503 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:22.493+0000 7fd5d7925a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:22.629 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:55:22.874 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:22.865+0000 7fd5d7925a40 -1 Falling back to public interface 2026-02-22T20:55:23.444 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:23.437+0000 7fd5d7925a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:55:23.790 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:55:23.948 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4265418873,v1:127.0.0.1:6803/4265418873] [v2:127.0.0.1:6804/4265418873,v1:127.0.0.1:6805/4265418873] exists,up 313477cb-fdcc-42dc-ae1a-2a329442519f 2026-02-22T20:55:23.952 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 27df203b-39f9-4cbc-a568-1816c719dcbf 2026-02-22T20:55:24.125 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:55:24.156 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:24.145+0000 7fe864301a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:24.157 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:24.149+0000 7fe864301a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:24.159 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:24.149+0000 7fe864301a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:24.159 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:24.149+0000 7fe864301a40 -1 bdev(0x559a4f44ec00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:55:24.159 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:24.149+0000 7fe864301a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-22T20:55:24.801 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:55:25.001 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:55:25.030 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:25.021+0000 7f31105c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:25.032 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:25.021+0000 7f31105c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:25.033 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:25.025+0000 7f31105c4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:25.164 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:55:25.410 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:25.401+0000 7f31105c4a40 -1 Falling back to public interface 2026-02-22T20:55:25.983 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:25.973+0000 7f31105c4a40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:55:26.330 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:55:26.487 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2571037530,v1:127.0.0.1:6811/2571037530] [v2:127.0.0.1:6812/2571037530,v1:127.0.0.1:6813/2571037530] exists,up 27df203b-39f9-4cbc-a568-1816c719dcbf 2026-02-22T20:55:26.493 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 e0c68e33-59d2-49ef-9ce4-8736c895335a 2026-02-22T20:55:26.655 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:55:26.694 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:26.685+0000 7fb3a8b9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:26.696 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:26.685+0000 7fb3a8b9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:26.697 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:26.689+0000 7fb3a8b9da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:26.697 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:26.689+0000 7fb3a8b9da40 -1 bdev(0x564beae12c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:55:26.697 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:26.689+0000 7fb3a8b9da40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-22T20:55:27.357 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:55:27.559 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:55:27.591 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:27.581+0000 7ff6a5c96a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:27.592 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:27.581+0000 7ff6a5c96a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:27.593 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:27.585+0000 7ff6a5c96a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:27.716 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:55:27.990 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:27.981+0000 7ff6a5c96a40 -1 Falling back to public interface 2026-02-22T20:55:28.614 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:28.605+0000 7ff6a5c96a40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:55:28.878 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:55:29.032 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3174130746,v1:127.0.0.1:6819/3174130746] [v2:127.0.0.1:6820/3174130746,v1:127.0.0.1:6821/3174130746] exists,up e0c68e33-59d2-49ef-9ce4-8736c895335a 2026-02-22T20:55:29.038 INFO:tasks.workunit.client.0.trial150.stdout:add osd3 c505bae6-357e-4952-8564-0949043d36d2 2026-02-22T20:55:29.210 INFO:tasks.workunit.client.0.trial150.stdout:3 2026-02-22T20:55:29.240 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:29.229+0000 7f42ddf17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:29.241 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:29.233+0000 7f42ddf17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:29.242 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:29.233+0000 7f42ddf17a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:29.243 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:29.233+0000 7f42ddf17a40 -1 bdev(0x55fda4274c00 td/osd-scrub-test/3/block) open stat got: (1) Operation not permitted 2026-02-22T20:55:29.243 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:29.233+0000 7f42ddf17a40 -1 bluestore(td/osd-scrub-test/3) _read_fsid unparsable uuid 2026-02-22T20:55:29.885 INFO:tasks.workunit.client.0.trial150.stdout:adding osd3 key to auth repository 2026-02-22T20:55:30.083 INFO:tasks.workunit.client.0.trial150.stdout:start osd.3 2026-02-22T20:55:30.113 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:30.105+0000 7f7401519a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:30.115 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:30.105+0000 7f7401519a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:30.116 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:30.109+0000 7f7401519a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:30.240 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:55:30.486 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:30.477+0000 7f7401519a40 -1 Falling back to public interface 2026-02-22T20:55:31.078 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:31.069+0000 7f7401519a40 -1 osd.3 0 log_to_monitors true 2026-02-22T20:55:31.400 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:55:31.557 INFO:tasks.workunit.client.0.trial150.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/839569101,v1:127.0.0.1:6827/839569101] [v2:127.0.0.1:6828/839569101,v1:127.0.0.1:6829/839569101] exists,up c505bae6-357e-4952-8564-0949043d36d2 2026-02-22T20:55:31.562 INFO:tasks.workunit.client.0.trial150.stdout:add osd4 e51e10f8-21fe-4f42-b19a-179709a23b97 2026-02-22T20:55:31.737 INFO:tasks.workunit.client.0.trial150.stdout:4 2026-02-22T20:55:31.767 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:31.757+0000 7f57ddad3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:31.768 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:31.757+0000 7f57ddad3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:31.769 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:31.761+0000 7f57ddad3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:31.770 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:31.761+0000 7f57ddad3a40 -1 bdev(0x55e7f3254c00 td/osd-scrub-test/4/block) open stat got: (1) Operation not permitted 2026-02-22T20:55:31.770 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:31.761+0000 7f57ddad3a40 -1 bluestore(td/osd-scrub-test/4) _read_fsid unparsable uuid 2026-02-22T20:55:32.424 INFO:tasks.workunit.client.0.trial150.stdout:adding osd4 key to auth repository 2026-02-22T20:55:32.620 INFO:tasks.workunit.client.0.trial150.stdout:start osd.4 2026-02-22T20:55:32.651 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:32.641+0000 7f3429695a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:32.652 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:32.645+0000 7f3429695a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:32.653 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:32.645+0000 7f3429695a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:55:32.776 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:55:33.042 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:33.033+0000 7f3429695a40 -1 Falling back to public interface 2026-02-22T20:55:33.672 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:55:33.661+0000 7f3429695a40 -1 osd.4 0 log_to_monitors true 2026-02-22T20:55:33.933 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:55:34.087 INFO:tasks.workunit.client.0.trial150.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3000510442,v1:127.0.0.1:6835/3000510442] [v2:127.0.0.1:6836/3000510442,v1:127.0.0.1:6837/3000510442] exists,up e51e10f8-21fe-4f42-b19a-179709a23b97 2026-02-22T20:55:34.292 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T20:55:35.948 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836493 2026-02-22T20:55:36.121 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705674 2026-02-22T20:55:36.285 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542152 2026-02-22T20:55:38.768 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.3 seq 77309411333 2026-02-22T20:55:38.938 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.4 seq 94489280515 2026-02-22T20:55:40.807 INFO:tasks.workunit.client.0.trial150.stdout:standard_scrub_cluster: dbg: test pool is test 1 2026-02-22T20:55:40.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:757: TEST_abort_periodic_for_operator: local poolid=1 2026-02-22T20:55:40.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:758: TEST_abort_periodic_for_operator: local poolname=test 2026-02-22T20:55:40.808 INFO:tasks.workunit.client.0.trial150.stdout:Pool: test : 1 2026-02-22T20:55:40.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:759: TEST_abort_periodic_for_operator: echo 'Pool: test : 1' 2026-02-22T20:55:40.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:762: TEST_abort_periodic_for_operator: local saved_echo_flag=x 2026-02-22T20:55:40.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:763: TEST_abort_periodic_for_operator: set +x 2026-02-22T20:55:40.809 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T20:55:40.810 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T20:55:40.810 INFO:tasks.workunit.client.0.trial150.stderr:320 bytes copied, 3.2751e-05 s, 9.8 MB/s 2026-02-22T20:55:46.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:776: TEST_abort_periodic_for_operator: declare -A pg_pr 2026-02-22T20:55:46.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:777: TEST_abort_periodic_for_operator: declare -A pg_ac 2026-02-22T20:55:46.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:778: TEST_abort_periodic_for_operator: declare -A pg_po 2026-02-22T20:55:46.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:779: TEST_abort_periodic_for_operator: build_pg_dicts td/osd-scrub-test pg_pr pg_ac pg_po - 2026-02-22T20:55:46.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:318: build_pg_dicts: local dir=td/osd-scrub-test 2026-02-22T20:55:46.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:319: build_pg_dicts: local -n pg_primary_dict=pg_pr 2026-02-22T20:55:46.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:320: build_pg_dicts: local -n pg_acting_dict=pg_ac 2026-02-22T20:55:46.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:321: build_pg_dicts: local -n pg_pool_dict=pg_po 2026-02-22T20:55:46.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:322: build_pg_dicts: local infile=- 2026-02-22T20:55:46.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:324: build_pg_dicts: local extr_dbg=0 2026-02-22T20:55:46.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:327: build_pg_dicts: local saved_echo_flag=x 2026-02-22T20:55:46.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:328: build_pg_dicts: set +x 2026-02-22T20:55:47.010 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs_brief 2026-02-22T20:55:47.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:780: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:781: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:47.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:782: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:47.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:785: TEST_abort_periodic_for_operator: wait_initial_scrubs pg_pr 2026-02-22T20:55:47.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:691: wait_initial_scrubs: local -n pg_to_prim_dict=pg_pr 2026-02-22T20:55:47.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:692: wait_initial_scrubs: local extr_dbg=1 2026-02-22T20:55:47.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:696: wait_initial_scrubs: ceph tell 'osd.*' config set osd_scrub_min_interval 7200 2026-02-22T20:55:47.764 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:47.764 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-22T20:55:47.764 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:47.772 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:47.772 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-22T20:55:47.772 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:47.779 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:55:47.779 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-22T20:55:47.779 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:47.786 INFO:tasks.workunit.client.0.trial150.stdout:osd.3: { 2026-02-22T20:55:47.786 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-22T20:55:47.786 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:47.794 INFO:tasks.workunit.client.0.trial150.stdout:osd.4: { 2026-02-22T20:55:47.794 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-22T20:55:47.794 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:47.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:697: wait_initial_scrubs: ceph tell 'osd.*' config set osd_deep_scrub_interval 14400 2026-02-22T20:55:47.876 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:47.877 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-22T20:55:47.877 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:47.884 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:47.884 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-22T20:55:47.884 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:47.891 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:55:47.891 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-22T20:55:47.891 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:47.898 INFO:tasks.workunit.client.0.trial150.stdout:osd.3: { 2026-02-22T20:55:47.898 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-22T20:55:47.898 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:47.905 INFO:tasks.workunit.client.0.trial150.stdout:osd.4: { 2026-02-22T20:55:47.906 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_deep_scrub_interval = '' " 2026-02-22T20:55:47.906 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:47.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:698: wait_initial_scrubs: ceph tell 'osd.*' config set osd_max_scrubs 32 2026-02-22T20:55:47.987 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:47.987 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_max_scrubs = '' " 2026-02-22T20:55:47.987 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:47.995 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:47.995 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_max_scrubs = '' " 2026-02-22T20:55:47.995 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.002 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:55:48.002 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_max_scrubs = '' " 2026-02-22T20:55:48.002 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.009 INFO:tasks.workunit.client.0.trial150.stdout:osd.3: { 2026-02-22T20:55:48.009 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_max_scrubs = '' " 2026-02-22T20:55:48.009 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.016 INFO:tasks.workunit.client.0.trial150.stdout:osd.4: { 2026-02-22T20:55:48.016 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_max_scrubs = '' " 2026-02-22T20:55:48.016 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:699: wait_initial_scrubs: ceph tell 'osd.*' config set osd_scrub_sleep 1 2026-02-22T20:55:48.090 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:48.090 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T20:55:48.090 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.097 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:48.097 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T20:55:48.097 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.104 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:55:48.104 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T20:55:48.104 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.112 INFO:tasks.workunit.client.0.trial150.stdout:osd.3: { 2026-02-22T20:55:48.112 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T20:55:48.112 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.119 INFO:tasks.workunit.client.0.trial150.stdout:osd.4: { 2026-02-22T20:55:48.119 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T20:55:48.120 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:700: wait_initial_scrubs: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 10 2026-02-22T20:55:48.193 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:48.193 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T20:55:48.193 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.200 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:48.200 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T20:55:48.201 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.208 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:55:48.208 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T20:55:48.208 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.215 INFO:tasks.workunit.client.0.trial150.stdout:osd.3: { 2026-02-22T20:55:48.215 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T20:55:48.215 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.222 INFO:tasks.workunit.client.0.trial150.stdout:osd.4: { 2026-02-22T20:55:48.222 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T20:55:48.222 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:701: wait_initial_scrubs: ceph tell 'osd.*' config set osd_scrub_chunk_max 10 2026-02-22T20:55:48.296 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:48.297 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-22T20:55:48.297 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.304 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:48.304 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-22T20:55:48.304 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.311 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:55:48.311 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-22T20:55:48.311 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.318 INFO:tasks.workunit.client.0.trial150.stdout:osd.3: { 2026-02-22T20:55:48.318 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-22T20:55:48.318 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.325 INFO:tasks.workunit.client.0.trial150.stdout:osd.4: { 2026-02-22T20:55:48.325 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-22T20:55:48.325 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:48.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:48.336 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.8 2026-02-22T20:55:48.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.8' 2026-02-22T20:55:48.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.8 scrub 2026-02-22T20:55:48.401 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:48.401 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:48.401 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:48.401 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:48.401 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:48.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:48.408 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.9 2026-02-22T20:55:48.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.9' 2026-02-22T20:55:48.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.9 scrub 2026-02-22T20:55:48.472 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:48.472 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:48.472 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:48.472 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:48.472 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:48.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:48.481 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.4 2026-02-22T20:55:48.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.4' 2026-02-22T20:55:48.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.4 scrub 2026-02-22T20:55:48.545 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:48.546 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:48.546 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:48.546 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:48.546 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:48.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:48.553 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.5 2026-02-22T20:55:48.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.5' 2026-02-22T20:55:48.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.5 scrub 2026-02-22T20:55:48.628 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:48.628 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:48.628 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:48.628 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:48.628 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:48.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:48.638 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.6 2026-02-22T20:55:48.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.6' 2026-02-22T20:55:48.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.6 scrub 2026-02-22T20:55:48.704 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:48.704 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:48.704 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:48.704 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:48.704 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:48.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:48.714 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.7 2026-02-22T20:55:48.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.7' 2026-02-22T20:55:48.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.7 scrub 2026-02-22T20:55:48.778 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:48.778 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:48.778 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:48.778 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:48.778 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:48.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:48.786 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.0 2026-02-22T20:55:48.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.0' 2026-02-22T20:55:48.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.0 scrub 2026-02-22T20:55:48.860 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:48.860 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:48.860 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:48.860 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:48.861 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:48.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:48.870 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.1 2026-02-22T20:55:48.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.1' 2026-02-22T20:55:48.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.1 scrub 2026-02-22T20:55:48.933 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:48.934 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:48.934 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:48.934 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:48.934 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:48.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:48.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:48.941 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.2 2026-02-22T20:55:48.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.2' 2026-02-22T20:55:48.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.2 scrub 2026-02-22T20:55:49.013 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:49.014 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:49.014 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:49.014 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:49.014 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:49.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:49.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:49.021 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.3 2026-02-22T20:55:49.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.3' 2026-02-22T20:55:49.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.3 scrub 2026-02-22T20:55:49.084 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:49.085 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:49.085 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:49.085 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:49.085 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:49.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:49.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:49.092 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.d 2026-02-22T20:55:49.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.d' 2026-02-22T20:55:49.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.d scrub 2026-02-22T20:55:49.165 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:49.165 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:49.166 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:49.166 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:49.166 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:49.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:49.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:49.173 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.e 2026-02-22T20:55:49.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.e' 2026-02-22T20:55:49.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.e scrub 2026-02-22T20:55:49.236 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:49.236 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:49.237 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:49.237 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:49.237 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:49.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:49.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:49.246 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.f 2026-02-22T20:55:49.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.f' 2026-02-22T20:55:49.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.f scrub 2026-02-22T20:55:49.310 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:49.310 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:49.310 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:49.310 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:49.310 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:49.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:49.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:49.318 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.a 2026-02-22T20:55:49.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.a' 2026-02-22T20:55:49.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.a scrub 2026-02-22T20:55:49.391 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:49.391 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:49.391 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:49.391 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:49.391 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:49.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:49.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:49.399 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.b 2026-02-22T20:55:49.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.b' 2026-02-22T20:55:49.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.b scrub 2026-02-22T20:55:49.471 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:49.471 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:49.472 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:49.472 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:49.472 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:49.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:703: wait_initial_scrubs: for pg in "${!pg_to_prim_dict[@]}" 2026-02-22T20:55:49.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:49.479 INFO:tasks.workunit.client.0.trial150.stdout:Scheduling initial scrub for 1.c 2026-02-22T20:55:49.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:704: wait_initial_scrubs: echo 'Scheduling initial scrub for 1.c' 2026-02-22T20:55:49.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:705: wait_initial_scrubs: ceph tell 1.c scrub 2026-02-22T20:55:49.551 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:49.551 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:55:49.551 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:55:49.551 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:55:49.551 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:49.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:708: wait_initial_scrubs: sleep 1 2026-02-22T20:55:50.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:709: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:50.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:709: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:50.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:710: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:50.703 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:50.712 INFO:tasks.workunit.client.0.trial150.stdout:[ 2026-02-22T20:55:50.712 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.712 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.f", 2026-02-22T20:55:50.712 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.712 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.712 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.712 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.e", 2026-02-22T20:55:50.712 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.712 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.712 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.713 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.d", 2026-02-22T20:55:50.713 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.713 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.713 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.713 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.0", 2026-02-22T20:55:50.713 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.713 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.713 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.713 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.1", 2026-02-22T20:55:50.713 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.713 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.713 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.c", 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.2", 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.3", 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.714 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.4", 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.5", 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.6", 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.7", 2026-02-22T20:55:50.715 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.8", 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.9", 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.a", 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.716 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:50.717 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:50.717 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.b", 2026-02-22T20:55:50.717 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:50.717 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:50.717 INFO:tasks.workunit.client.0.trial150.stdout:] 2026-02-22T20:55:50.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:712: wait_initial_scrubs: tout=20 2026-02-22T20:55:50.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 20 -gt 0 ']' 2026-02-22T20:55:50.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-22T20:55:51.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:51.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:51.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:51.861 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:51.871 INFO:tasks.workunit.client.0.trial150.stdout:[ 2026-02-22T20:55:51.871 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:51.871 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.f", 2026-02-22T20:55:51.871 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:51.871 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:51.871 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:51.871 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.e", 2026-02-22T20:55:51.871 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:51.871 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.0", 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.c", 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.3", 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:51.872 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:51.873 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.7", 2026-02-22T20:55:51.873 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:51.873 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:51.873 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:51.873 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.a", 2026-02-22T20:55:51.873 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:51.873 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:51.873 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:51.873 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.b", 2026-02-22T20:55:51.873 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:51.873 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:51.873 INFO:tasks.workunit.client.0.trial150.stdout:] 2026-02-22T20:55:51.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:51.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:51.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-22T20:55:52.020 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:52.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=34 2026-02-22T20:55:52.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 34 -le 1 ']' 2026-02-22T20:55:52.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=8 2026-02-22T20:55:52.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 8 PGs to finish initial scrubs (timeout 20)' 2026-02-22T20:55:52.031 INFO:tasks.workunit.client.0.trial150.stdout:Still waiting for 8 PGs to finish initial scrubs (timeout 20) 2026-02-22T20:55:52.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=19 2026-02-22T20:55:52.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 19 -gt 0 ']' 2026-02-22T20:55:52.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-22T20:55:53.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:53.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:53.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:53.177 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:53.186 INFO:tasks.workunit.client.0.trial150.stdout:[ 2026-02-22T20:55:53.186 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:53.186 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.f", 2026-02-22T20:55:53.186 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:53.187 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:53.187 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:53.187 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.e", 2026-02-22T20:55:53.187 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:53.187 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:53.187 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:53.187 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.0", 2026-02-22T20:55:53.187 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:53.187 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:53.187 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:53.187 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.c", 2026-02-22T20:55:53.187 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.3", 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.7", 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.a", 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:53.188 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:53.189 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:53.189 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.b", 2026-02-22T20:55:53.189 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:53.189 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:53.189 INFO:tasks.workunit.client.0.trial150.stdout:] 2026-02-22T20:55:53.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:53.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:53.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-22T20:55:53.335 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:53.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=34 2026-02-22T20:55:53.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 34 -le 1 ']' 2026-02-22T20:55:53.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=8 2026-02-22T20:55:53.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 8 PGs to finish initial scrubs (timeout 19)' 2026-02-22T20:55:53.344 INFO:tasks.workunit.client.0.trial150.stdout:Still waiting for 8 PGs to finish initial scrubs (timeout 19) 2026-02-22T20:55:53.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=18 2026-02-22T20:55:53.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 18 -gt 0 ']' 2026-02-22T20:55:53.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-22T20:55:54.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:54.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:54.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:54.496 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:54.504 INFO:tasks.workunit.client.0.trial150.stdout:[ 2026-02-22T20:55:54.504 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:54.504 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.e", 2026-02-22T20:55:54.504 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:54.504 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:54.505 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:54.505 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.a", 2026-02-22T20:55:54.505 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:54.505 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T20:55:54.505 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:54.505 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.b", 2026-02-22T20:55:54.505 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:54.505 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:54.505 INFO:tasks.workunit.client.0.trial150.stdout:] 2026-02-22T20:55:54.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:54.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:54.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-22T20:55:54.655 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:54.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=14 2026-02-22T20:55:54.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 14 -le 1 ']' 2026-02-22T20:55:54.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=3 2026-02-22T20:55:54.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 3 PGs to finish initial scrubs (timeout 18)' 2026-02-22T20:55:54.665 INFO:tasks.workunit.client.0.trial150.stdout:Still waiting for 3 PGs to finish initial scrubs (timeout 18) 2026-02-22T20:55:54.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=17 2026-02-22T20:55:54.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 17 -gt 0 ']' 2026-02-22T20:55:54.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-22T20:55:55.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:55.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:55.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:55.809 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:55.818 INFO:tasks.workunit.client.0.trial150.stdout:[ 2026-02-22T20:55:55.819 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:55.819 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.b", 2026-02-22T20:55:55.819 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:55.819 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:55.819 INFO:tasks.workunit.client.0.trial150.stdout:] 2026-02-22T20:55:55.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:55.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:55.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-22T20:55:55.970 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:55.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=6 2026-02-22T20:55:55.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 6 -le 1 ']' 2026-02-22T20:55:55.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=1 2026-02-22T20:55:55.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 1 PGs to finish initial scrubs (timeout 17)' 2026-02-22T20:55:55.980 INFO:tasks.workunit.client.0.trial150.stdout:Still waiting for 1 PGs to finish initial scrubs (timeout 17) 2026-02-22T20:55:55.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=16 2026-02-22T20:55:55.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 16 -gt 0 ']' 2026-02-22T20:55:55.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-22T20:55:56.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:56.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:56.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:57.133 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:57.142 INFO:tasks.workunit.client.0.trial150.stdout:[ 2026-02-22T20:55:57.142 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:55:57.142 INFO:tasks.workunit.client.0.trial150.stdout: "pgid": "1.b", 2026-02-22T20:55:57.142 INFO:tasks.workunit.client.0.trial150.stdout: "last_scrub_duration": 0 2026-02-22T20:55:57.142 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:55:57.142 INFO:tasks.workunit.client.0.trial150.stdout:] 2026-02-22T20:55:57.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:57.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:57.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-22T20:55:57.293 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:57.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=6 2026-02-22T20:55:57.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 6 -le 1 ']' 2026-02-22T20:55:57.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:723: wait_initial_scrubs: not_done=1 2026-02-22T20:55:57.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:724: wait_initial_scrubs: echo 'Still waiting for 1 PGs to finish initial scrubs (timeout 16)' 2026-02-22T20:55:57.304 INFO:tasks.workunit.client.0.trial150.stdout:Still waiting for 1 PGs to finish initial scrubs (timeout 16) 2026-02-22T20:55:57.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:725: wait_initial_scrubs: tout=15 2026-02-22T20:55:57.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:713: wait_initial_scrubs: '[' 15 -gt 0 ']' 2026-02-22T20:55:57.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:714: wait_initial_scrubs: sleep 1 2026-02-22T20:55:58.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: (( extr_dbg >= 1 )) 2026-02-22T20:55:58.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:715: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:58.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:716: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:58.453 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:58.462 INFO:tasks.workunit.client.0.trial150.stdout:[] 2026-02-22T20:55:58.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:58.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:58.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: wc -l 2026-02-22T20:55:58.612 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:58.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:718: wait_initial_scrubs: not_done=1 2026-02-22T20:55:58.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:720: wait_initial_scrubs: '[' 1 -le 1 ']' 2026-02-22T20:55:58.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:721: wait_initial_scrubs: break 2026-02-22T20:55:58.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:727: wait_initial_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T20:55:58.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:728: wait_initial_scrubs: jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' 2026-02-22T20:55:58.769 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:58.779 INFO:tasks.workunit.client.0.trial150.stdout:[] 2026-02-22T20:55:58.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:729: wait_initial_scrubs: (( tout == 0 )) 2026-02-22T20:55:58.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:730: wait_initial_scrubs: return 0 2026-02-22T20:55:58.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:788: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_max_scrubs 1 2026-02-22T20:55:58.844 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:58.844 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_max_scrubs = '' " 2026-02-22T20:55:58.844 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:58.852 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:58.852 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_max_scrubs = '' " 2026-02-22T20:55:58.852 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:58.859 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:55:58.859 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_max_scrubs = '' " 2026-02-22T20:55:58.859 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:58.867 INFO:tasks.workunit.client.0.trial150.stdout:osd.3: { 2026-02-22T20:55:58.867 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_max_scrubs = '' " 2026-02-22T20:55:58.867 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:58.874 INFO:tasks.workunit.client.0.trial150.stdout:osd.4: { 2026-02-22T20:55:58.874 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_max_scrubs = '' " 2026-02-22T20:55:58.874 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:58.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:789: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_stats_update_period_not_scrubbing 1 2026-02-22T20:55:58.948 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:58.948 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-22T20:55:58.948 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:58.955 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:58.955 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-22T20:55:58.955 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:58.962 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:55:58.962 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-22T20:55:58.963 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:58.970 INFO:tasks.workunit.client.0.trial150.stdout:osd.3: { 2026-02-22T20:55:58.970 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-22T20:55:58.970 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:58.977 INFO:tasks.workunit.client.0.trial150.stdout:osd.4: { 2026-02-22T20:55:58.977 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_stats_update_period_not_scrubbing = '' " 2026-02-22T20:55:58.977 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:58.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:792: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_scrub_sleep 3 2026-02-22T20:55:59.048 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:59.048 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T20:55:59.048 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.055 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:59.055 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T20:55:59.055 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.062 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:55:59.062 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T20:55:59.063 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.070 INFO:tasks.workunit.client.0.trial150.stdout:osd.3: { 2026-02-22T20:55:59.070 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T20:55:59.070 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.077 INFO:tasks.workunit.client.0.trial150.stdout:osd.4: { 2026-02-22T20:55:59.077 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T20:55:59.077 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:793: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 2 2026-02-22T20:55:59.152 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:59.152 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T20:55:59.152 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.159 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:59.160 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T20:55:59.160 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.167 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:55:59.167 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T20:55:59.167 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.174 INFO:tasks.workunit.client.0.trial150.stdout:osd.3: { 2026-02-22T20:55:59.174 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T20:55:59.174 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.181 INFO:tasks.workunit.client.0.trial150.stdout:osd.4: { 2026-02-22T20:55:59.181 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T20:55:59.182 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:794: TEST_abort_periodic_for_operator: ceph tell 'osd.*' config set osd_scrub_chunk_max 2 2026-02-22T20:55:59.254 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:55:59.254 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-22T20:55:59.254 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.261 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:55:59.261 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-22T20:55:59.261 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.268 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:55:59.268 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-22T20:55:59.268 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.275 INFO:tasks.workunit.client.0.trial150.stdout:osd.3: { 2026-02-22T20:55:59.275 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-22T20:55:59.275 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.282 INFO:tasks.workunit.client.0.trial150.stdout:osd.4: { 2026-02-22T20:55:59.282 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_chunk_max = '' " 2026-02-22T20:55:59.283 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:795: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:55:59.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:798: TEST_abort_periodic_for_operator: local pg2= 2026-02-22T20:55:59.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:799: TEST_abort_periodic_for_operator: for pg1 in "${!pg_pr[@]}" 2026-02-22T20:55:59.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:800: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:59.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:801: TEST_abort_periodic_for_operator: [[ 1.8 == \1\.\8 ]] 2026-02-22T20:55:59.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:802: TEST_abort_periodic_for_operator: continue 2026-02-22T20:55:59.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:800: TEST_abort_periodic_for_operator: for pg in "${!pg_pr[@]}" 2026-02-22T20:55:59.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:801: TEST_abort_periodic_for_operator: [[ 1.9 == \1\.\8 ]] 2026-02-22T20:55:59.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:804: TEST_abort_periodic_for_operator: [[ 1 == \1 ]] 2026-02-22T20:55:59.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:805: TEST_abort_periodic_for_operator: count_common_active 1.9 1.8 pg_ac 2026-02-22T20:55:59.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:367: count_common_active: local pg1=1.9 2026-02-22T20:55:59.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:368: count_common_active: local pg2=1.8 2026-02-22T20:55:59.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:369: count_common_active: local -n pg_acting_dict=pg_ac 2026-02-22T20:55:59.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:371: count_common_active: a1=('1' '4' '2') 2026-02-22T20:55:59.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:371: count_common_active: local -a a1 2026-02-22T20:55:59.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:372: count_common_active: a2=('1' '4' '3') 2026-02-22T20:55:59.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:372: count_common_active: local -a a2 2026-02-22T20:55:59.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:374: count_common_active: local -i cnt=0 2026-02-22T20:55:59.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:375: count_common_active: for i in "${a1[@]}" 2026-02-22T20:55:59.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-22T20:55:59.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 1 -eq 1 ]] 2026-02-22T20:55:59.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:378: count_common_active: cnt=1 2026-02-22T20:55:59.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-22T20:55:59.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 1 -eq 4 ]] 2026-02-22T20:55:59.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-22T20:55:59.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 1 -eq 3 ]] 2026-02-22T20:55:59.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:375: count_common_active: for i in "${a1[@]}" 2026-02-22T20:55:59.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-22T20:55:59.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 4 -eq 1 ]] 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 4 -eq 4 ]] 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:378: count_common_active: cnt=2 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 4 -eq 3 ]] 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:375: count_common_active: for i in "${a1[@]}" 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 2 -eq 1 ]] 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 2 -eq 4 ]] 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:376: count_common_active: for j in "${a2[@]}" 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:377: count_common_active: [[ 2 -eq 3 ]] 2026-02-22T20:55:59.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:383: count_common_active: printf %d 2 2026-02-22T20:55:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:805: TEST_abort_periodic_for_operator: local -i common=2 2026-02-22T20:55:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:806: TEST_abort_periodic_for_operator: [[ 2 -gt 1 ]] 2026-02-22T20:55:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:807: TEST_abort_periodic_for_operator: pg2=1.9 2026-02-22T20:55:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:808: TEST_abort_periodic_for_operator: break 2 2026-02-22T20:55:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:814: TEST_abort_periodic_for_operator: [[ -z 1.9 ]] 2026-02-22T20:55:59.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:820: TEST_abort_periodic_for_operator: ceph tell osd.1 config set osd_max_scrubs 2 2026-02-22T20:55:59.361 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:55:59.362 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_max_scrubs = '' " 2026-02-22T20:55:59.362 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:55:59.369 INFO:tasks.workunit.client.0.trial150.stdout:The two PGs to manipulate are 1.8 and 1.9 2026-02-22T20:55:59.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:821: TEST_abort_periodic_for_operator: echo 'The two PGs to manipulate are 1.8 and 1.9' 2026-02-22T20:55:59.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:823: TEST_abort_periodic_for_operator: set_query_debug 1.8 2026-02-22T20:55:59.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:417: set_query_debug: local pgid=1.8 2026-02-22T20:55:59.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: ceph pg dump pgs_brief 2026-02-22T20:55:59.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: awk -v 'pg=^1.8' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2026-02-22T20:55:59.513 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs_brief 2026-02-22T20:55:59.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: local prim_osd=1 2026-02-22T20:55:59.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:421: set_query_debug: echo 'Setting scrub debug data. Primary for 1.8 is 1' 2026-02-22T20:55:59.524 INFO:tasks.workunit.client.0.trial150.stdout:Setting scrub debug data. Primary for 1.8 is 1 2026-02-22T20:55:59.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: get_asok_path osd.1 2026-02-22T20:55:59.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:55:59.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:55:59.525 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:55:59.525 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:55:59.525 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:55:59.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-osd.1.asok 2026-02-22T20:55:59.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: CEPH_ARGS= 2026-02-22T20:55:59.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.516403/ceph-osd.1.asok scrubdebug 1.8 set sessions 2026-02-22T20:55:59.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:826: TEST_abort_periodic_for_operator: local is_act 2026-02-22T20:55:59.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:827: TEST_abort_periodic_for_operator: seq 1 3 2026-02-22T20:55:59.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:827: TEST_abort_periodic_for_operator: for i in $( seq 1 3 ) 2026-02-22T20:55:59.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:829: TEST_abort_periodic_for_operator: ceph pg 1.8 query 2026-02-22T20:55:59.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:829: TEST_abort_periodic_for_operator: jq .scrubber.active 2026-02-22T20:55:59.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:829: TEST_abort_periodic_for_operator: is_act=false 2026-02-22T20:55:59.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:830: TEST_abort_periodic_for_operator: [[ false = \f\a\l\s\e ]] 2026-02-22T20:55:59.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:831: TEST_abort_periodic_for_operator: break 2026-02-22T20:55:59.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:836: TEST_abort_periodic_for_operator: ceph pg dump pgs 2026-02-22T20:55:59.807 INFO:tasks.workunit.client.0.trial150.stdout:{"success":true}PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2026-02-22T20:55:59.807 INFO:tasks.workunit.client.0.trial150.stdout:1.f 19 0 0 0 0 6080 0 0 19 0 19 active+clean 2026-02-22T20:55:52.217114+0000 26'19 26:46 [3,0,4] 3 [3,0,4] 3 26'19 2026-02-22T20:55:52.217065+0000 0'0 2026-02-22T20:55:34.286701+0000 0 2 periodic scrub scheduled @ 2026-02-22T22:55:52.217065+0000 19 0 2026-02-22T20:55:59.807 INFO:tasks.workunit.client.0.trial150.stdout:1.e 20 0 0 0 0 6400 0 0 20 0 20 active+clean 2026-02-22T20:55:53.743257+0000 26'20 26:47 [1,4,2] 1 [1,4,2] 1 26'20 2026-02-22T20:55:53.743165+0000 0'0 2026-02-22T20:55:34.286701+0000 0 2 periodic scrub scheduled @ 2026-02-22T22:55:53.743165+0000 20 0 2026-02-22T20:55:59.807 INFO:tasks.workunit.client.0.trial150.stdout:1.d 14 0 0 0 0 4480 0 0 14 0 14 active+clean 2026-02-22T20:55:50.402623+0000 26'14 26:41 [2,4,1] 2 [2,4,1] 2 26'14 2026-02-22T20:55:50.402571+0000 0'0 2026-02-22T20:55:34.286701+0000 0 1 periodic scrub scheduled @ 2026-02-22T22:55:50.402571+0000 14 0 2026-02-22T20:55:59.807 INFO:tasks.workunit.client.0.trial150.stdout:1.0 22 0 0 0 0 7040 0 0 22 0 22 active+clean 2026-02-22T20:55:52.775908+0000 26'22 26:55 [1,0,2] 1 [1,0,2] 1 26'22 2026-02-22T20:55:52.775836+0000 0'0 2026-02-22T20:55:34.286701+0000 0 2 periodic scrub scheduled @ 2026-02-22T22:55:52.775836+0000 22 0 2026-02-22T20:55:59.807 INFO:tasks.workunit.client.0.trial150.stdout:1.1 12 0 0 0 0 3840 0 0 12 0 12 active+clean 2026-02-22T20:55:50.711686+0000 26'12 26:43 [4,3,0] 4 [4,3,0] 4 26'12 2026-02-22T20:55:50.711625+0000 0'0 2026-02-22T20:55:34.286701+0000 0 1 periodic scrub scheduled @ 2026-02-22T22:55:50.711625+0000 12 0 2026-02-22T20:55:59.808 INFO:tasks.workunit.client.0.trial150.stdout:1.c 15 0 0 0 0 4800 0 0 15 0 15 active+clean 2026-02-22T20:55:53.178124+0000 26'15 26:43 [3,0,4] 3 [3,0,4] 3 26'15 2026-02-22T20:55:53.178073+0000 0'0 2026-02-22T20:55:34.286701+0000 0 1 periodic scrub scheduled @ 2026-02-22T22:55:53.178073+0000 15 0 2026-02-22T20:55:59.808 INFO:tasks.workunit.client.0.trial150.stdout:1.2 15 0 0 0 0 4800 0 0 15 0 15 active+clean 2026-02-22T20:55:50.274090+0000 26'15 26:46 [0,4,1] 0 [0,4,1] 0 26'15 2026-02-22T20:55:50.274045+0000 0'0 2026-02-22T20:55:34.286701+0000 0 1 periodic scrub scheduled @ 2026-02-22T22:55:50.274045+0000 15 0 2026-02-22T20:55:59.808 INFO:tasks.workunit.client.0.trial150.stdout:1.3 19 0 0 0 0 6080 0 0 19 0 19 active+clean 2026-02-22T20:55:51.729597+0000 26'19 26:49 [4,1,2] 4 [4,1,2] 4 26'19 2026-02-22T20:55:51.729546+0000 0'0 2026-02-22T20:55:34.286701+0000 0 2 periodic scrub scheduled @ 2026-02-22T22:55:51.729546+0000 19 0 2026-02-22T20:55:59.808 INFO:tasks.workunit.client.0.trial150.stdout:1.4 15 0 0 0 0 4800 0 0 15 0 15 active+clean 2026-02-22T20:55:50.252500+0000 26'15 26:44 [3,0,1] 3 [3,0,1] 3 26'15 2026-02-22T20:55:50.252436+0000 0'0 2026-02-22T20:55:34.286701+0000 0 2 periodic scrub scheduled @ 2026-02-22T22:55:50.252436+0000 15 0 2026-02-22T20:55:59.808 INFO:tasks.workunit.client.0.trial150.stdout:1.5 26 0 0 0 0 8320 0 0 26 0 26 active+clean 2026-02-22T20:55:49.668377+0000 26'26 26:55 [4,2,3] 4 [4,2,3] 4 26'26 2026-02-22T20:55:49.668293+0000 0'0 2026-02-22T20:55:34.286701+0000 0 2 periodic scrub scheduled @ 2026-02-22T22:55:49.668293+0000 26 0 2026-02-22T20:55:59.808 INFO:tasks.workunit.client.0.trial150.stdout:1.6 12 0 0 0 0 3840 0 0 12 0 12 active+clean 2026-02-22T20:55:51.209617+0000 26'12 26:41 [3,4,2] 3 [3,4,2] 3 26'12 2026-02-22T20:55:51.209558+0000 0'0 2026-02-22T20:55:34.286701+0000 0 2 periodic scrub scheduled @ 2026-02-22T22:55:51.209558+0000 12 0 2026-02-22T20:55:59.808 INFO:tasks.workunit.client.0.trial150.stdout:1.7 11 0 0 0 0 3520 0 0 11 0 11 active+clean 2026-02-22T20:55:51.731321+0000 26'11 26:39 [1,3,0] 1 [1,3,0] 1 26'11 2026-02-22T20:55:51.731282+0000 0'0 2026-02-22T20:55:34.286701+0000 0 1 periodic scrub scheduled @ 2026-02-22T22:55:51.731282+0000 11 0 2026-02-22T20:55:59.808 INFO:tasks.workunit.client.0.trial150.stdout:1.8 28 0 0 0 0 8960 0 0 28 0 28 active+clean 2026-02-22T20:55:49.733190+0000 26'28 26:57 [1,4,3] 1 [1,4,3] 1 26'28 2026-02-22T20:55:49.733134+0000 0'0 2026-02-22T20:55:34.286701+0000 0 2 periodic scrub scheduled @ 2026-02-22T22:55:49.733134+0000 28 0 2026-02-22T20:55:59.808 INFO:tasks.workunit.client.0.trial150.stdout:1.9 8 0 0 0 0 2560 0 0 8 0 8 active+clean 2026-02-22T20:55:50.752076+0000 26'8 26:36 [1,4,2] 1 [1,4,2] 1 26'8 2026-02-22T20:55:50.752036+0000 0'0 2026-02-22T20:55:34.286701+0000 0 1 periodic scrub scheduled @ 2026-02-22T22:55:50.752036+0000 8 0 2026-02-22T20:55:59.808 INFO:tasks.workunit.client.0.trial150.stdout:1.a 9 0 0 0 0 2880 0 0 9 0 9 active+clean 2026-02-22T20:55:54.730122+0000 26'9 26:37 [1,2,4] 1 [1,2,4] 1 26'9 2026-02-22T20:55:54.730059+0000 0'0 2026-02-22T20:55:34.286701+0000 0 1 periodic scrub scheduled @ 2026-02-22T22:55:54.730059+0000 9 0 2026-02-22T20:55:59.808 INFO:tasks.workunit.client.0.trial150.stdout:1.b 11 0 0 0 0 3520 0 0 11 0 11 active+clean 2026-02-22T20:55:55.772465+0000 26'11 26:39 [1,0,2] 1 [1,0,2] 1 26'11 2026-02-22T20:55:55.772405+0000 0'0 2026-02-22T20:55:34.286701+0000 0 1 periodic scrub scheduled @ 2026-02-22T22:55:55.772405+0000 11 0 2026-02-22T20:55:59.808 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:55:59.809 INFO:tasks.workunit.client.0.trial150.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2026-02-22T20:55:59.809 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:55:59.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:837: TEST_abort_periodic_for_operator: [[ false != \f\a\l\s\e ]] 2026-02-22T20:55:59.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:842: TEST_abort_periodic_for_operator: sleep 0.5 2026-02-22T20:56:00.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:844: TEST_abort_periodic_for_operator: echo 'Initiating a periodic scrub of 1.8' 2026-02-22T20:56:00.318 INFO:tasks.workunit.client.0.trial150.stdout:Initiating a periodic scrub of 1.8 2026-02-22T20:56:00.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:845: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:56:00.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:846: TEST_abort_periodic_for_operator: ceph tell 1.8 schedule-deep-scrub 2026-02-22T20:56:00.382 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:56:00.382 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:56:00.382 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:56:00.382 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-22T14:31:50.377733+0000" 2026-02-22T20:56:00.382 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:56:00.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:847: TEST_abort_periodic_for_operator: sleep 1 2026-02-22T20:56:01.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:848: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:56:01.391 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:850: TEST_abort_periodic_for_operator: seq 1 14 2026-02-22T20:56:01.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:850: TEST_abort_periodic_for_operator: for i in $( seq 1 14 ) 2026-02-22T20:56:01.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:852: TEST_abort_periodic_for_operator: sleep 0.5 2026-02-22T20:56:01.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:853: TEST_abort_periodic_for_operator: ceph pg 1.8 query 2026-02-22T20:56:01.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:853: TEST_abort_periodic_for_operator: jq .scrubber 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:853: TEST_abort_periodic_for_operator: stt='{ 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "active": true, 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "epoch_start": "23", 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "start": "1:10000000::::0", 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "end": "MIN", 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "max_end": "MIN", 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "subset_last_update": "0'\''0", 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "deep": true, 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "req_scrub": false, 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "auto_repair": false, 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "check_repair": false, 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "deep_scrub_on_error": false, 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "priority": 5, 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "shallow_errors": 0, 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "deep_errors": 0, 2026-02-22T20:56:01.965 INFO:tasks.workunit.client.0.trial150.stderr: "fixed": 0, 2026-02-22T20:56:01.966 INFO:tasks.workunit.client.0.trial150.stderr: "waiting_on_whom": [], 2026-02-22T20:56:01.966 INFO:tasks.workunit.client.0.trial150.stderr: "schedule": "scrubbing", 2026-02-22T20:56:01.966 INFO:tasks.workunit.client.0.trial150.stderr: "is_reserving_replicas": false, 2026-02-22T20:56:01.966 INFO:tasks.workunit.client.0.trial150.stderr: "test_sequence": 13606916 2026-02-22T20:56:01.966 INFO:tasks.workunit.client.0.trial150.stderr:}' 2026-02-22T20:56:01.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:854: TEST_abort_periodic_for_operator: jq .active 2026-02-22T20:56:01.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:854: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"23",' '"start":' '"1:10000000::::0",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 5, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' false, '"test_sequence":' 13606916 '}' 2026-02-22T20:56:01.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:854: TEST_abort_periodic_for_operator: is_active=true 2026-02-22T20:56:01.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:855: TEST_abort_periodic_for_operator: jq .is_reserving_replicas 2026-02-22T20:56:01.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:855: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"23",' '"start":' '"1:10000000::::0",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 5, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' false, '"test_sequence":' 13606916 '}' 2026-02-22T20:56:01.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:855: TEST_abort_periodic_for_operator: is_reserving_replicas=false 2026-02-22T20:56:01.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:856: TEST_abort_periodic_for_operator: [[ true = \t\r\u\e ]] 2026-02-22T20:56:01.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:856: TEST_abort_periodic_for_operator: [[ false = \f\a\l\s\e ]] 2026-02-22T20:56:01.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:857: TEST_abort_periodic_for_operator: break 2026-02-22T20:56:01.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:861: TEST_abort_periodic_for_operator: [[ true != \t\r\u\e ]] 2026-02-22T20:56:01.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:861: TEST_abort_periodic_for_operator: [[ false != \f\a\l\s\e ]] 2026-02-22T20:56:01.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:866: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:56:01.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:873: TEST_abort_periodic_for_operator: (( extr_dbg >= 1 )) 2026-02-22T20:56:01.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:873: TEST_abort_periodic_for_operator: ceph tell 'osd.*' dump_scrub_reservations --format=json-pretty 2026-02-22T20:56:02.049 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:56:02.049 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T20:56:02.050 INFO:tasks.workunit.client.0.trial150.stdout: "scrubs_local": 0, 2026-02-22T20:56:02.050 INFO:tasks.workunit.client.0.trial150.stdout: "osd_max_scrubs": 1, 2026-02-22T20:56:02.050 INFO:tasks.workunit.client.0.trial150.stdout: "remote_scrub_reservations": { 2026-02-22T20:56:02.050 INFO:tasks.workunit.client.0.trial150.stdout: "max_allowed": 1, 2026-02-22T20:56:02.050 INFO:tasks.workunit.client.0.trial150.stdout: "min_priority": 0, 2026-02-22T20:56:02.050 INFO:tasks.workunit.client.0.trial150.stdout: "queues": [], 2026-02-22T20:56:02.050 INFO:tasks.workunit.client.0.trial150.stdout: "in_progress": [] 2026-02-22T20:56:02.050 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:56:02.050 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:56:02.057 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:56:02.057 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T20:56:02.057 INFO:tasks.workunit.client.0.trial150.stdout: "scrubs_local": 1, 2026-02-22T20:56:02.057 INFO:tasks.workunit.client.0.trial150.stdout: "osd_max_scrubs": 2, 2026-02-22T20:56:02.057 INFO:tasks.workunit.client.0.trial150.stdout: "remote_scrub_reservations": { 2026-02-22T20:56:02.057 INFO:tasks.workunit.client.0.trial150.stdout: "max_allowed": 2, 2026-02-22T20:56:02.057 INFO:tasks.workunit.client.0.trial150.stdout: "min_priority": 0, 2026-02-22T20:56:02.057 INFO:tasks.workunit.client.0.trial150.stdout: "queues": [], 2026-02-22T20:56:02.057 INFO:tasks.workunit.client.0.trial150.stdout: "in_progress": [] 2026-02-22T20:56:02.057 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:56:02.058 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:56:02.065 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:56:02.065 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T20:56:02.065 INFO:tasks.workunit.client.0.trial150.stdout: "scrubs_local": 0, 2026-02-22T20:56:02.065 INFO:tasks.workunit.client.0.trial150.stdout: "osd_max_scrubs": 1, 2026-02-22T20:56:02.065 INFO:tasks.workunit.client.0.trial150.stdout: "remote_scrub_reservations": { 2026-02-22T20:56:02.065 INFO:tasks.workunit.client.0.trial150.stdout: "max_allowed": 1, 2026-02-22T20:56:02.065 INFO:tasks.workunit.client.0.trial150.stdout: "min_priority": 0, 2026-02-22T20:56:02.066 INFO:tasks.workunit.client.0.trial150.stdout: "queues": [], 2026-02-22T20:56:02.066 INFO:tasks.workunit.client.0.trial150.stdout: "in_progress": [] 2026-02-22T20:56:02.066 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:56:02.066 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout:osd.3: { 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: "scrubs_local": 0, 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: "osd_max_scrubs": 1, 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: "remote_scrub_reservations": { 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: "max_allowed": 1, 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: "min_priority": 0, 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: "queues": [], 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: "in_progress": [ 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: "item": "1.8", 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: "prio": 0, 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: "can_preempt": false 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:56:02.073 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout:osd.4: { 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: "scrubs_local": 0, 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: "osd_max_scrubs": 1, 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: "remote_scrub_reservations": { 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: "max_allowed": 1, 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: "min_priority": 0, 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: "queues": [], 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: "in_progress": [ 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: "item": "1.8", 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: "prio": 0, 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: "can_preempt": false 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T20:56:02.081 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:56:02.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:876: TEST_abort_periodic_for_operator: set_query_debug 1.9 2026-02-22T20:56:02.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:417: set_query_debug: local pgid=1.9 2026-02-22T20:56:02.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: ceph pg dump pgs_brief 2026-02-22T20:56:02.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: awk -v 'pg=^1.9' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2026-02-22T20:56:02.233 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs_brief 2026-02-22T20:56:02.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: local prim_osd=1 2026-02-22T20:56:02.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:421: set_query_debug: echo 'Setting scrub debug data. Primary for 1.9 is 1' 2026-02-22T20:56:02.243 INFO:tasks.workunit.client.0.trial150.stdout:Setting scrub debug data. Primary for 1.9 is 1 2026-02-22T20:56:02.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: get_asok_path osd.1 2026-02-22T20:56:02.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T20:56:02.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T20:56:02.244 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:56:02.244 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:56:02.244 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:56:02.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-osd.1.asok 2026-02-22T20:56:02.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: CEPH_ARGS= 2026-02-22T20:56:02.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.516403/ceph-osd.1.asok scrubdebug 1.9 set sessions 2026-02-22T20:56:02.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:877: TEST_abort_periodic_for_operator: ceph tell 1.9 schedule-deep-scrub 2026-02-22T20:56:02.386 INFO:tasks.workunit.client.0.trial150.stdout:{"success":true}{ 2026-02-22T20:56:02.387 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:56:02.387 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:56:02.387 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-22T14:31:52.381612+0000" 2026-02-22T20:56:02.387 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:56:02.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:878: TEST_abort_periodic_for_operator: sleep 0.5 2026-02-22T20:56:02.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:879: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:56:02.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:880: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:56:02.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:881: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:56:02.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:882: TEST_abort_periodic_for_operator: sleep 1 2026-02-22T20:56:03.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:883: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:56:03.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:884: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:56:03.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:885: TEST_abort_periodic_for_operator: (( extr_dbg >= 2 )) 2026-02-22T20:56:03.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:888: TEST_abort_periodic_for_operator: ceph pg 1.9 query 2026-02-22T20:56:03.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:888: TEST_abort_periodic_for_operator: jq .scrubber 2026-02-22T20:56:03.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:888: TEST_abort_periodic_for_operator: local 'stt2={ 2026-02-22T20:56:03.969 INFO:tasks.workunit.client.0.trial150.stderr: "active": true, 2026-02-22T20:56:03.969 INFO:tasks.workunit.client.0.trial150.stderr: "epoch_start": "23", 2026-02-22T20:56:03.969 INFO:tasks.workunit.client.0.trial150.stderr: "start": "MIN", 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "end": "MIN", 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "max_end": "MIN", 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "subset_last_update": "0'\''0", 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "deep": true, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "req_scrub": false, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "auto_repair": false, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "check_repair": false, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "deep_scrub_on_error": false, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "priority": 5, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "shallow_errors": 0, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "deep_errors": 0, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "fixed": 0, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "waiting_on_whom": [], 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "schedule": "scrubbing", 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "is_reserving_replicas": true, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "osd_to_respond": 4, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "duration_seconds": 2, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "requested_in_order": 2, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "num_to_reserve": 2, 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr: "test_sequence": 12034051 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr:}' 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:889: TEST_abort_periodic_for_operator: local pg2_is_reserving 2026-02-22T20:56:03.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:890: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"23",' '"start":' '"MIN",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 5, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' true, '"osd_to_respond":' 4, '"duration_seconds":' 2, '"requested_in_order":' 2, '"num_to_reserve":' 2, '"test_sequence":' 12034051 '}' 2026-02-22T20:56:03.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:890: TEST_abort_periodic_for_operator: jq .is_reserving_replicas 2026-02-22T20:56:03.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:890: TEST_abort_periodic_for_operator: pg2_is_reserving=true 2026-02-22T20:56:03.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:891: TEST_abort_periodic_for_operator: [[ true != \t\r\u\e ]] 2026-02-22T20:56:03.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:899: TEST_abort_periodic_for_operator: echo 'Instructing 1.9 to perform a high-priority scrub' 2026-02-22T20:56:03.980 INFO:tasks.workunit.client.0.trial150.stdout:Instructing 1.9 to perform a high-priority scrub 2026-02-22T20:56:03.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:900: TEST_abort_periodic_for_operator: ceph tell 1.9 scrub 2026-02-22T20:56:04.045 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:56:04.045 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T20:56:04.045 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T20:56:04.045 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T20:56:04.045 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:56:04.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:901: TEST_abort_periodic_for_operator: seq 1 10 2026-02-22T20:56:04.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:901: TEST_abort_periodic_for_operator: for i in $( seq 1 10 ) 2026-02-22T20:56:04.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:903: TEST_abort_periodic_for_operator: sleep 0.5 2026-02-22T20:56:04.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:904: TEST_abort_periodic_for_operator: ceph pg 1.9 query 2026-02-22T20:56:04.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:904: TEST_abort_periodic_for_operator: jq .scrubber 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:904: TEST_abort_periodic_for_operator: stt2='{ 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr: "active": true, 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr: "epoch_start": "23", 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr: "start": "MIN", 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr: "end": "MIN", 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr: "max_end": "MIN", 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr: "subset_last_update": "0'\''0", 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr: "deep": true, 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr: "req_scrub": false, 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr: "auto_repair": false, 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr: "check_repair": false, 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr: "deep_scrub_on_error": false, 2026-02-22T20:56:04.625 INFO:tasks.workunit.client.0.trial150.stderr: "priority": 0, 2026-02-22T20:56:04.626 INFO:tasks.workunit.client.0.trial150.stderr: "shallow_errors": 0, 2026-02-22T20:56:04.626 INFO:tasks.workunit.client.0.trial150.stderr: "deep_errors": 0, 2026-02-22T20:56:04.626 INFO:tasks.workunit.client.0.trial150.stderr: "fixed": 0, 2026-02-22T20:56:04.626 INFO:tasks.workunit.client.0.trial150.stderr: "waiting_on_whom": [], 2026-02-22T20:56:04.626 INFO:tasks.workunit.client.0.trial150.stderr: "schedule": "scrubbing", 2026-02-22T20:56:04.626 INFO:tasks.workunit.client.0.trial150.stderr: "is_reserving_replicas": false, 2026-02-22T20:56:04.626 INFO:tasks.workunit.client.0.trial150.stderr: "test_sequence": 12034052 2026-02-22T20:56:04.626 INFO:tasks.workunit.client.0.trial150.stderr:}' 2026-02-22T20:56:04.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:905: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"23",' '"start":' '"MIN",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 0, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' false, '"test_sequence":' 12034052 '}' 2026-02-22T20:56:04.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:905: TEST_abort_periodic_for_operator: jq .active 2026-02-22T20:56:04.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:905: TEST_abort_periodic_for_operator: pg2_is_active=true 2026-02-22T20:56:04.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:906: TEST_abort_periodic_for_operator: jq .is_reserving_replicas 2026-02-22T20:56:04.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:906: TEST_abort_periodic_for_operator: echo '{' '"active":' true, '"epoch_start":' '"23",' '"start":' '"MIN",' '"end":' '"MIN",' '"max_end":' '"MIN",' '"subset_last_update":' '"0'\''0",' '"deep":' true, '"req_scrub":' false, '"auto_repair":' false, '"check_repair":' false, '"deep_scrub_on_error":' false, '"priority":' 0, '"shallow_errors":' 0, '"deep_errors":' 0, '"fixed":' 0, '"waiting_on_whom":' '[],' '"schedule":' '"scrubbing",' '"is_reserving_replicas":' false, '"test_sequence":' 12034052 '}' 2026-02-22T20:56:04.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:906: TEST_abort_periodic_for_operator: pg2_is_reserving=false 2026-02-22T20:56:04.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:907: TEST_abort_periodic_for_operator: [[ true = \t\r\u\e ]] 2026-02-22T20:56:04.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:907: TEST_abort_periodic_for_operator: [[ false != \t\r\u\e ]] 2026-02-22T20:56:04.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:908: TEST_abort_periodic_for_operator: break 2026-02-22T20:56:04.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:913: TEST_abort_periodic_for_operator: [[ true != \t\r\u\e ]] 2026-02-22T20:56:04.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:913: TEST_abort_periodic_for_operator: [[ false = \t\r\u\e ]] 2026-02-22T20:56:04.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:917: TEST_abort_periodic_for_operator: echo Done 2026-02-22T20:56:04.646 INFO:tasks.workunit.client.0.trial150.stdout:Done 2026-02-22T20:56:04.646 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Teardown Test TEST_abort_periodic_for_operator ------------------ 2026-02-22T20:56:04.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_abort_periodic_for_operator ------------------' 2026-02-22T20:56:04.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-22T20:56:04.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T20:56:04.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:56:04.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T20:56:04.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:56:04.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:56:04.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:56:04.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:56:04.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:56:04.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:56:04.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:56:04.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:56:04.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:56:04.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:56:04.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:56:04.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:56:04.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:56:04.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:56:04.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:56:04.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:56:04.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:56:04.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:56:04.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T20:56:04.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:56:04.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:56:04.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:56:04.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T20:56:04.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:56:04.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:56:04.992 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Complete Test TEST_abort_periodic_for_operator ------------------ 2026-02-22T20:56:04.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_abort_periodic_for_operator ------------------' 2026-02-22T20:56:04.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-22T20:56:04.992 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Prepare Test TEST_deep_scrub_abort ------------------- 2026-02-22T20:56:04.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_deep_scrub_abort -------------------' 2026-02-22T20:56:04.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-22T20:56:04.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-22T20:56:04.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-22T20:56:04.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T20:56:04.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T20:56:04.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T20:56:04.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T20:56:04.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T20:56:04.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T20:56:04.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T20:56:04.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T20:56:04.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T20:56:04.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T20:56:04.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T20:56:04.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T20:56:04.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T20:56:04.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T20:56:04.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T20:56:04.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:56:04.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T20:56:04.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T20:56:05.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T20:56:05.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T20:56:05.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T20:56:05.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T20:56:05.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T20:56:05.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:56:05.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:56:05.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T20:56:05.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T20:56:05.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T20:56:05.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-22T20:56:05.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T20:56:05.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:56:05.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:56:05.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.516403 2026-02-22T20:56:05.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T20:56:05.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T20:56:05.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T20:56:05.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-22T20:56:05.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_deep_scrub_abort -----------------------' 2026-02-22T20:56:05.009 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Run Test TEST_deep_scrub_abort ----------------------- 2026-02-22T20:56:05.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_deep_scrub_abort td/osd-scrub-test 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:409: TEST_deep_scrub_abort: local dir=td/osd-scrub-test 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:410: TEST_deep_scrub_abort: _scrub_abort td/osd-scrub-test deep-scrub 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:293: _scrub_abort: local dir=td/osd-scrub-test 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: local poolname=test 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:295: _scrub_abort: local OSDS=3 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:296: _scrub_abort: local objects=1000 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:297: _scrub_abort: local type=deep-scrub 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:299: _scrub_abort: TESTDATA=testdata.516403 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: test deep-scrub = scrub 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:305: _scrub_abort: stopscrub=nodeep-scrub 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:306: _scrub_abort: check=nodeep_scrub 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:309: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-22T20:56:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-22T20:56:05.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T20:56:05.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:56:05.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:56:05.031 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:56:05.031 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:56:05.031 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:56:05.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T20:56:05.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-22T20:56:05.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T20:56:05.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T20:56:05.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:56:05.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:56:05.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T20:56:05.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T20:56:05.063 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:56:05.063 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:56:05.063 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:56:05.064 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:56:05.064 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:56:05.064 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:56:05.064 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T20:56:05.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:56:05.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get fsid 2026-02-22T20:56:05.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T20:56:05.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T20:56:05.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T20:56:05.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T20:56:05.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T20:56:05.120 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T20:56:05.120 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T20:56:05.120 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T20:56:05.121 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T20:56:05.121 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:56:05.121 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:56:05.121 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T20:56:05.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T20:56:05.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get mon_host 2026-02-22T20:56:05.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-22T20:56:05.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-22T20:56:05.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T20:56:05.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T20:56:05.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T20:56:05.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-22T20:56:05.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T20:56:05.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T20:56:05.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:56:05.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:56:05.271 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:56:05.272 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:56:05.272 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:56:05.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T20:56:05.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T20:56:05.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-22T20:56:05.292 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: expr 3 - 1 2026-02-22T20:56:05.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: seq 0 2 2026-02-22T20:56:05.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:56:05.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:56:05.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:56:05.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:56:05.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:56:05.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-22T20:56:05.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-22T20:56:05.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:56:05.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ed3f003c-9214-4c0e-887e-00ff6caa63bd 2026-02-22T20:56:05.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ed3f003c-9214-4c0e-887e-00ff6caa63bd' 2026-02-22T20:56:05.298 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 ed3f003c-9214-4c0e-887e-00ff6caa63bd 2026-02-22T20:56:05.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:56:05.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBlbZtpbLweEhAAA/iHxRky2ytXadGEVuARdA== 2026-02-22T20:56:05.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBlbZtpbLweEhAAA/iHxRky2ytXadGEVuARdA=="}' 2026-02-22T20:56:05.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ed3f003c-9214-4c0e-887e-00ff6caa63bd -i td/osd-scrub-test/0/new.json 2026-02-22T20:56:05.399 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:56:05.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-22T20:56:05.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBlbZtpbLweEhAAA/iHxRky2ytXadGEVuARdA== --osd-uuid ed3f003c-9214-4c0e-887e-00ff6caa63bd 2026-02-22T20:56:05.424 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:05.412+0000 7f28ae522a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:05.425 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:05.416+0000 7f28ae522a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:05.426 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:05.416+0000 7f28ae522a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:05.427 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:05.416+0000 7f28ae522a40 -1 bdev(0x55bf99af2c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-22T20:56:05.427 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:05.416+0000 7f28ae522a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-22T20:56:06.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-22T20:56:06.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:56:06.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T20:56:06.200 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T20:56:06.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:56:06.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T20:56:06.404 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T20:56:06.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-22T20:56:06.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:56:06.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:56:06.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:56:06.430 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:06.420+0000 7fc8f8e50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:06.432 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:06.420+0000 7fc8f8e50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:06.433 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:06.424+0000 7fc8f8e50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:06.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T20:56:06.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:56:06.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T20:56:06.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:56:06.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:56:06.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:56:06.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:56:06.564 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:56:06.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:56:06.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:56:06.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:56:06.822 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:06.812+0000 7fc8f8e50a40 -1 Falling back to public interface 2026-02-22T20:56:07.452 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:07.440+0000 7fc8f8e50a40 -1 osd.0 0 log_to_monitors true 2026-02-22T20:56:07.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:56:07.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:56:07.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:56:07.725 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:56:07.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:56:07.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T20:56:07.884 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4185489986,v1:127.0.0.1:6803/4185489986] [v2:127.0.0.1:6804/4185489986,v1:127.0.0.1:6805/4185489986] exists,up ed3f003c-9214-4c0e-887e-00ff6caa63bd 2026-02-22T20:56:07.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:56:07.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:56:07.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:56:07.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:56:07.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-22T20:56:07.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T20:56:07.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:56:07.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T20:56:07.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:56:07.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-22T20:56:07.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T20:56:07.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:56:07.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:56:07.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:56:07.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-22T20:56:07.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-22T20:56:07.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:56:07.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:56:07.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T20:56:07.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:56:07.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:56:07.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:56:07.894 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:56:07.895 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:56:07.895 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:56:07.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T20:56:07.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T20:56:07.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:56:07.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:56:07.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:56:07.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T20:56:07.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T20:56:07.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:56:07.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:56:07.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:56:07.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:56:07.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:56:07.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-22T20:56:07.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-22T20:56:07.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:56:07.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2cfe4245-57e3-4343-91d4-fe610cee8fd3 2026-02-22T20:56:07.898 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 2cfe4245-57e3-4343-91d4-fe610cee8fd3 2026-02-22T20:56:07.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2cfe4245-57e3-4343-91d4-fe610cee8fd3' 2026-02-22T20:56:07.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:56:07.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBnbZtp4fxLNRAASHivB+DiYDazI6n4kG4ckg== 2026-02-22T20:56:07.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBnbZtp4fxLNRAASHivB+DiYDazI6n4kG4ckg=="}' 2026-02-22T20:56:07.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2cfe4245-57e3-4343-91d4-fe610cee8fd3 -i td/osd-scrub-test/1/new.json 2026-02-22T20:56:08.065 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:56:08.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-22T20:56:08.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBnbZtp4fxLNRAASHivB+DiYDazI6n4kG4ckg== --osd-uuid 2cfe4245-57e3-4343-91d4-fe610cee8fd3 2026-02-22T20:56:08.091 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:08.080+0000 7f472cefea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:08.093 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:08.084+0000 7f472cefea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:08.094 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:08.084+0000 7f472cefea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:08.094 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:08.084+0000 7f472cefea40 -1 bdev(0x55ce68b12c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-22T20:56:08.094 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:08.084+0000 7f472cefea40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-22T20:56:08.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-22T20:56:08.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:56:08.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T20:56:08.725 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T20:56:08.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:56:08.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T20:56:08.931 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T20:56:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-22T20:56:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:56:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:56:08.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:56:08.955 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:08.944+0000 7ff52961ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:08.957 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:08.948+0000 7ff52961ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:08.958 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:08.948+0000 7ff52961ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:09.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T20:56:09.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:56:09.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T20:56:09.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:56:09.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:56:09.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:56:09.090 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:56:09.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:56:09.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:56:09.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:56:09.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:56:09.370 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:09.360+0000 7ff52961ca40 -1 Falling back to public interface 2026-02-22T20:56:09.911 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:09.900+0000 7ff52961ca40 -1 osd.1 0 log_to_monitors true 2026-02-22T20:56:10.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:56:10.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:56:10.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:56:10.419 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:56:10.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:56:10.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T20:56:10.420 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/643777510,v1:127.0.0.1:6811/643777510] [v2:127.0.0.1:6812/643777510,v1:127.0.0.1:6813/643777510] exists,up 2cfe4245-57e3-4343-91d4-fe610cee8fd3 2026-02-22T20:56:10.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:56:10.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:56:10.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:56:10.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T20:56:10.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-22T20:56:10.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T20:56:10.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T20:56:10.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T20:56:10.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T20:56:10.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-22T20:56:10.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T20:56:10.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T20:56:10.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T20:56:10.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T20:56:10.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-22T20:56:10.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-22T20:56:10.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T20:56:10.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T20:56:10.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T20:56:10.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T20:56:10.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T20:56:10.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T20:56:10.427 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T20:56:10.427 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T20:56:10.427 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T20:56:10.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T20:56:10.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T20:56:10.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T20:56:10.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T20:56:10.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T20:56:10.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T20:56:10.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T20:56:10.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T20:56:10.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T20:56:10.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T20:56:10.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T20:56:10.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T20:56:10.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-22T20:56:10.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-22T20:56:10.428 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T20:56:10.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=66634184-2b8c-4f06-afd2-789b24e1a5ec 2026-02-22T20:56:10.429 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 66634184-2b8c-4f06-afd2-789b24e1a5ec 2026-02-22T20:56:10.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 66634184-2b8c-4f06-afd2-789b24e1a5ec' 2026-02-22T20:56:10.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T20:56:10.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBqbZtpa8XhGBAA2C2lXmMJwPjc8N+PNQSyrg== 2026-02-22T20:56:10.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBqbZtpa8XhGBAA2C2lXmMJwPjc8N+PNQSyrg=="}' 2026-02-22T20:56:10.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 66634184-2b8c-4f06-afd2-789b24e1a5ec -i td/osd-scrub-test/2/new.json 2026-02-22T20:56:10.584 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T20:56:10.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-22T20:56:10.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQBqbZtpa8XhGBAA2C2lXmMJwPjc8N+PNQSyrg== --osd-uuid 66634184-2b8c-4f06-afd2-789b24e1a5ec 2026-02-22T20:56:10.612 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:10.600+0000 7f763b8a4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:10.613 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:10.604+0000 7f763b8a4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:10.614 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:10.604+0000 7f763b8a4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:10.615 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:10.604+0000 7f763b8a4a40 -1 bdev(0x55f425ba8c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-22T20:56:10.615 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:10.604+0000 7f763b8a4a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-22T20:56:11.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-22T20:56:11.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T20:56:11.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T20:56:11.269 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T20:56:11.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T20:56:11.478 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T20:56:11.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T20:56:11.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-22T20:56:11.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T20:56:11.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T20:56:11.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T20:56:11.503 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:11.492+0000 7f7004aeda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:11.505 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:11.496+0000 7f7004aeda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:11.506 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:11.496+0000 7f7004aeda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T20:56:11.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T20:56:11.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T20:56:11.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T20:56:11.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T20:56:11.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T20:56:11.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:56:11.639 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T20:56:11.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T20:56:11.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:56:11.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:56:11.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T20:56:11.950 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:11.940+0000 7f7004aeda40 -1 Falling back to public interface 2026-02-22T20:56:12.643 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T20:56:12.632+0000 7f7004aeda40 -1 osd.2 0 log_to_monitors true 2026-02-22T20:56:12.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T20:56:12.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T20:56:12.797 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T20:56:12.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T20:56:12.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T20:56:12.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T20:56:12.952 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2366473767,v1:127.0.0.1:6819/2366473767] [v2:127.0.0.1:6820/2366473767,v1:127.0.0.1:6821/2366473767] exists,up 66634184-2b8c-4f06-afd2-789b24e1a5ec 2026-02-22T20:56:12.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T20:56:12.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T20:56:12.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T20:56:12.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:325: _scrub_abort: create_pool test 1 1 2026-02-22T20:56:12.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-22T20:56:13.155 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T20:56:13.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T20:56:14.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:326: _scrub_abort: wait_for_clean 2026-02-22T20:56:14.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T20:56:14.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T20:56:14.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T20:56:14.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T20:56:14.169 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T20:56:14.169 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T20:56:14.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T20:56:14.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T20:56:14.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T20:56:14.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T20:56:14.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T20:56:14.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T20:56:14.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T20:56:14.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:56:14.280 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:56:14.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:56:14.442 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:56:14.442 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:56:14.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:56:14.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:14.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:56:14.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-22T20:56:14.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-22T20:56:14.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-22T20:56:14.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:14.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:56:14.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-22T20:56:14.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-22T20:56:14.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-38654705669' 2026-02-22T20:56:14.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:14.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:56:14.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574850 2026-02-22T20:56:14.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574850 2026-02-22T20:56:14.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-38654705669 2-55834574850' 2026-02-22T20:56:14.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:14.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-22T20:56:14.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:14.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:56:14.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-22T20:56:14.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:14.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-22T20:56:14.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-22T20:56:14.663 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836487 2026-02-22T20:56:14.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:14.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T20:56:14.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:56:15.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:56:15.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:15.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836487 2026-02-22T20:56:15.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:15.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-22T20:56:15.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:15.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:56:15.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-22T20:56:15.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:15.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-22T20:56:15.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-22T20:56:15.999 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705669 2026-02-22T20:56:15.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:56:16.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705670 -lt 38654705669 2026-02-22T20:56:16.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:16.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574850 2026-02-22T20:56:16.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:16.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:56:16.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574850 2026-02-22T20:56:16.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:16.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574850 2026-02-22T20:56:16.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574850' 2026-02-22T20:56:16.172 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574850 2026-02-22T20:56:16.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:56:16.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574851 -lt 55834574850 2026-02-22T20:56:16.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T20:56:16.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:56:16.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:56:16.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T20:56:16.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T20:56:16.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T20:56:16.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T20:56:16.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T20:56:16.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T20:56:16.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T20:56:16.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T20:56:16.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T20:56:16.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T20:56:16.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T20:56:16.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T20:56:16.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T20:56:16.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T20:56:16.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T20:56:16.882 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: ceph osd dump 2026-02-22T20:56:16.882 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: grep '^pool.*['\'']test['\'']' 2026-02-22T20:56:16.882 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: awk '{ print $2 }' 2026-02-22T20:56:17.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: poolid=1 2026-02-22T20:56:17.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:329: _scrub_abort: dd if=/dev/urandom of=testdata.516403 bs=1032 count=1 2026-02-22T20:56:17.036 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T20:56:17.036 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T20:56:17.036 INFO:tasks.workunit.client.0.trial150.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.6709e-05 s, 28.1 MB/s 2026-02-22T20:56:17.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: seq 1 1000 2026-02-22T20:56:17.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj1 testdata.516403 2026-02-22T20:56:17.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj2 testdata.516403 2026-02-22T20:56:17.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj3 testdata.516403 2026-02-22T20:56:17.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj4 testdata.516403 2026-02-22T20:56:17.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj5 testdata.516403 2026-02-22T20:56:17.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj6 testdata.516403 2026-02-22T20:56:17.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj7 testdata.516403 2026-02-22T20:56:17.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj8 testdata.516403 2026-02-22T20:56:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj9 testdata.516403 2026-02-22T20:56:17.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj10 testdata.516403 2026-02-22T20:56:17.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj11 testdata.516403 2026-02-22T20:56:17.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj12 testdata.516403 2026-02-22T20:56:17.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj13 testdata.516403 2026-02-22T20:56:17.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj14 testdata.516403 2026-02-22T20:56:17.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj15 testdata.516403 2026-02-22T20:56:17.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj16 testdata.516403 2026-02-22T20:56:17.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj17 testdata.516403 2026-02-22T20:56:17.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj18 testdata.516403 2026-02-22T20:56:17.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj19 testdata.516403 2026-02-22T20:56:17.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj20 testdata.516403 2026-02-22T20:56:17.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj21 testdata.516403 2026-02-22T20:56:17.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj22 testdata.516403 2026-02-22T20:56:17.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj23 testdata.516403 2026-02-22T20:56:17.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj24 testdata.516403 2026-02-22T20:56:17.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj25 testdata.516403 2026-02-22T20:56:17.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj26 testdata.516403 2026-02-22T20:56:17.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj27 testdata.516403 2026-02-22T20:56:17.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj28 testdata.516403 2026-02-22T20:56:17.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj29 testdata.516403 2026-02-22T20:56:17.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj30 testdata.516403 2026-02-22T20:56:17.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj31 testdata.516403 2026-02-22T20:56:17.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj32 testdata.516403 2026-02-22T20:56:17.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj33 testdata.516403 2026-02-22T20:56:17.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj34 testdata.516403 2026-02-22T20:56:17.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj35 testdata.516403 2026-02-22T20:56:17.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj36 testdata.516403 2026-02-22T20:56:17.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj37 testdata.516403 2026-02-22T20:56:17.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj38 testdata.516403 2026-02-22T20:56:17.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj39 testdata.516403 2026-02-22T20:56:17.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj40 testdata.516403 2026-02-22T20:56:17.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj41 testdata.516403 2026-02-22T20:56:17.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:17.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj42 testdata.516403 2026-02-22T20:56:18.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj43 testdata.516403 2026-02-22T20:56:18.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj44 testdata.516403 2026-02-22T20:56:18.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj45 testdata.516403 2026-02-22T20:56:18.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj46 testdata.516403 2026-02-22T20:56:18.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj47 testdata.516403 2026-02-22T20:56:18.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj48 testdata.516403 2026-02-22T20:56:18.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj49 testdata.516403 2026-02-22T20:56:18.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj50 testdata.516403 2026-02-22T20:56:18.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj51 testdata.516403 2026-02-22T20:56:18.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj52 testdata.516403 2026-02-22T20:56:18.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj53 testdata.516403 2026-02-22T20:56:18.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj54 testdata.516403 2026-02-22T20:56:18.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj55 testdata.516403 2026-02-22T20:56:18.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj56 testdata.516403 2026-02-22T20:56:18.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj57 testdata.516403 2026-02-22T20:56:18.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj58 testdata.516403 2026-02-22T20:56:18.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj59 testdata.516403 2026-02-22T20:56:18.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj60 testdata.516403 2026-02-22T20:56:18.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj61 testdata.516403 2026-02-22T20:56:18.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj62 testdata.516403 2026-02-22T20:56:18.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj63 testdata.516403 2026-02-22T20:56:18.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj64 testdata.516403 2026-02-22T20:56:18.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj65 testdata.516403 2026-02-22T20:56:18.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj66 testdata.516403 2026-02-22T20:56:18.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj67 testdata.516403 2026-02-22T20:56:18.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj68 testdata.516403 2026-02-22T20:56:18.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj69 testdata.516403 2026-02-22T20:56:18.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj70 testdata.516403 2026-02-22T20:56:18.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj71 testdata.516403 2026-02-22T20:56:18.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj72 testdata.516403 2026-02-22T20:56:18.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj73 testdata.516403 2026-02-22T20:56:18.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj74 testdata.516403 2026-02-22T20:56:18.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj75 testdata.516403 2026-02-22T20:56:18.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj76 testdata.516403 2026-02-22T20:56:18.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj77 testdata.516403 2026-02-22T20:56:18.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj78 testdata.516403 2026-02-22T20:56:18.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj79 testdata.516403 2026-02-22T20:56:18.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj80 testdata.516403 2026-02-22T20:56:18.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj81 testdata.516403 2026-02-22T20:56:18.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj82 testdata.516403 2026-02-22T20:56:18.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj83 testdata.516403 2026-02-22T20:56:18.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj84 testdata.516403 2026-02-22T20:56:18.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:18.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj85 testdata.516403 2026-02-22T20:56:19.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj86 testdata.516403 2026-02-22T20:56:19.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj87 testdata.516403 2026-02-22T20:56:19.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj88 testdata.516403 2026-02-22T20:56:19.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj89 testdata.516403 2026-02-22T20:56:19.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj90 testdata.516403 2026-02-22T20:56:19.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj91 testdata.516403 2026-02-22T20:56:19.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj92 testdata.516403 2026-02-22T20:56:19.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj93 testdata.516403 2026-02-22T20:56:19.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj94 testdata.516403 2026-02-22T20:56:19.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj95 testdata.516403 2026-02-22T20:56:19.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj96 testdata.516403 2026-02-22T20:56:19.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj97 testdata.516403 2026-02-22T20:56:19.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj98 testdata.516403 2026-02-22T20:56:19.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj99 testdata.516403 2026-02-22T20:56:19.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj100 testdata.516403 2026-02-22T20:56:19.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj101 testdata.516403 2026-02-22T20:56:19.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj102 testdata.516403 2026-02-22T20:56:19.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj103 testdata.516403 2026-02-22T20:56:19.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj104 testdata.516403 2026-02-22T20:56:19.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj105 testdata.516403 2026-02-22T20:56:19.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj106 testdata.516403 2026-02-22T20:56:19.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj107 testdata.516403 2026-02-22T20:56:19.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj108 testdata.516403 2026-02-22T20:56:19.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj109 testdata.516403 2026-02-22T20:56:19.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj110 testdata.516403 2026-02-22T20:56:19.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj111 testdata.516403 2026-02-22T20:56:19.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj112 testdata.516403 2026-02-22T20:56:19.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj113 testdata.516403 2026-02-22T20:56:19.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj114 testdata.516403 2026-02-22T20:56:19.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj115 testdata.516403 2026-02-22T20:56:19.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj116 testdata.516403 2026-02-22T20:56:19.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj117 testdata.516403 2026-02-22T20:56:19.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj118 testdata.516403 2026-02-22T20:56:19.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj119 testdata.516403 2026-02-22T20:56:19.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj120 testdata.516403 2026-02-22T20:56:19.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj121 testdata.516403 2026-02-22T20:56:19.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj122 testdata.516403 2026-02-22T20:56:19.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj123 testdata.516403 2026-02-22T20:56:19.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj124 testdata.516403 2026-02-22T20:56:19.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj125 testdata.516403 2026-02-22T20:56:19.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj126 testdata.516403 2026-02-22T20:56:19.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj127 testdata.516403 2026-02-22T20:56:19.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj128 testdata.516403 2026-02-22T20:56:19.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:19.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj129 testdata.516403 2026-02-22T20:56:20.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj130 testdata.516403 2026-02-22T20:56:20.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj131 testdata.516403 2026-02-22T20:56:20.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj132 testdata.516403 2026-02-22T20:56:20.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj133 testdata.516403 2026-02-22T20:56:20.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj134 testdata.516403 2026-02-22T20:56:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj135 testdata.516403 2026-02-22T20:56:20.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj136 testdata.516403 2026-02-22T20:56:20.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj137 testdata.516403 2026-02-22T20:56:20.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj138 testdata.516403 2026-02-22T20:56:20.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj139 testdata.516403 2026-02-22T20:56:20.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj140 testdata.516403 2026-02-22T20:56:20.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj141 testdata.516403 2026-02-22T20:56:20.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj142 testdata.516403 2026-02-22T20:56:20.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj143 testdata.516403 2026-02-22T20:56:20.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj144 testdata.516403 2026-02-22T20:56:20.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj145 testdata.516403 2026-02-22T20:56:20.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj146 testdata.516403 2026-02-22T20:56:20.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj147 testdata.516403 2026-02-22T20:56:20.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj148 testdata.516403 2026-02-22T20:56:20.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj149 testdata.516403 2026-02-22T20:56:20.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj150 testdata.516403 2026-02-22T20:56:20.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj151 testdata.516403 2026-02-22T20:56:20.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj152 testdata.516403 2026-02-22T20:56:20.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj153 testdata.516403 2026-02-22T20:56:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj154 testdata.516403 2026-02-22T20:56:20.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj155 testdata.516403 2026-02-22T20:56:20.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj156 testdata.516403 2026-02-22T20:56:20.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj157 testdata.516403 2026-02-22T20:56:20.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj158 testdata.516403 2026-02-22T20:56:20.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj159 testdata.516403 2026-02-22T20:56:20.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj160 testdata.516403 2026-02-22T20:56:20.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj161 testdata.516403 2026-02-22T20:56:20.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj162 testdata.516403 2026-02-22T20:56:20.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj163 testdata.516403 2026-02-22T20:56:20.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj164 testdata.516403 2026-02-22T20:56:20.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj165 testdata.516403 2026-02-22T20:56:20.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj166 testdata.516403 2026-02-22T20:56:20.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj167 testdata.516403 2026-02-22T20:56:20.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj168 testdata.516403 2026-02-22T20:56:20.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj169 testdata.516403 2026-02-22T20:56:20.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj170 testdata.516403 2026-02-22T20:56:20.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:20.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj171 testdata.516403 2026-02-22T20:56:21.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj172 testdata.516403 2026-02-22T20:56:21.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj173 testdata.516403 2026-02-22T20:56:21.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj174 testdata.516403 2026-02-22T20:56:21.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj175 testdata.516403 2026-02-22T20:56:21.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj176 testdata.516403 2026-02-22T20:56:21.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj177 testdata.516403 2026-02-22T20:56:21.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj178 testdata.516403 2026-02-22T20:56:21.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj179 testdata.516403 2026-02-22T20:56:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj180 testdata.516403 2026-02-22T20:56:21.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj181 testdata.516403 2026-02-22T20:56:21.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj182 testdata.516403 2026-02-22T20:56:21.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj183 testdata.516403 2026-02-22T20:56:21.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj184 testdata.516403 2026-02-22T20:56:21.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj185 testdata.516403 2026-02-22T20:56:21.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj186 testdata.516403 2026-02-22T20:56:21.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj187 testdata.516403 2026-02-22T20:56:21.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj188 testdata.516403 2026-02-22T20:56:21.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj189 testdata.516403 2026-02-22T20:56:21.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj190 testdata.516403 2026-02-22T20:56:21.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj191 testdata.516403 2026-02-22T20:56:21.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj192 testdata.516403 2026-02-22T20:56:21.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj193 testdata.516403 2026-02-22T20:56:21.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj194 testdata.516403 2026-02-22T20:56:21.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj195 testdata.516403 2026-02-22T20:56:21.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj196 testdata.516403 2026-02-22T20:56:21.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj197 testdata.516403 2026-02-22T20:56:21.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj198 testdata.516403 2026-02-22T20:56:21.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj199 testdata.516403 2026-02-22T20:56:21.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj200 testdata.516403 2026-02-22T20:56:21.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj201 testdata.516403 2026-02-22T20:56:21.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj202 testdata.516403 2026-02-22T20:56:21.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj203 testdata.516403 2026-02-22T20:56:21.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj204 testdata.516403 2026-02-22T20:56:21.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj205 testdata.516403 2026-02-22T20:56:21.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj206 testdata.516403 2026-02-22T20:56:21.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj207 testdata.516403 2026-02-22T20:56:21.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj208 testdata.516403 2026-02-22T20:56:21.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj209 testdata.516403 2026-02-22T20:56:21.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj210 testdata.516403 2026-02-22T20:56:21.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj211 testdata.516403 2026-02-22T20:56:21.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj212 testdata.516403 2026-02-22T20:56:21.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj213 testdata.516403 2026-02-22T20:56:21.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:21.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj214 testdata.516403 2026-02-22T20:56:22.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj215 testdata.516403 2026-02-22T20:56:22.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj216 testdata.516403 2026-02-22T20:56:22.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj217 testdata.516403 2026-02-22T20:56:22.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj218 testdata.516403 2026-02-22T20:56:22.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj219 testdata.516403 2026-02-22T20:56:22.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj220 testdata.516403 2026-02-22T20:56:22.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj221 testdata.516403 2026-02-22T20:56:22.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj222 testdata.516403 2026-02-22T20:56:22.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj223 testdata.516403 2026-02-22T20:56:22.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj224 testdata.516403 2026-02-22T20:56:22.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj225 testdata.516403 2026-02-22T20:56:22.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj226 testdata.516403 2026-02-22T20:56:22.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj227 testdata.516403 2026-02-22T20:56:22.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj228 testdata.516403 2026-02-22T20:56:22.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj229 testdata.516403 2026-02-22T20:56:22.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj230 testdata.516403 2026-02-22T20:56:22.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj231 testdata.516403 2026-02-22T20:56:22.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj232 testdata.516403 2026-02-22T20:56:22.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj233 testdata.516403 2026-02-22T20:56:22.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj234 testdata.516403 2026-02-22T20:56:22.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj235 testdata.516403 2026-02-22T20:56:22.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj236 testdata.516403 2026-02-22T20:56:22.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj237 testdata.516403 2026-02-22T20:56:22.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj238 testdata.516403 2026-02-22T20:56:22.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj239 testdata.516403 2026-02-22T20:56:22.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj240 testdata.516403 2026-02-22T20:56:22.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj241 testdata.516403 2026-02-22T20:56:22.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj242 testdata.516403 2026-02-22T20:56:22.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj243 testdata.516403 2026-02-22T20:56:22.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj244 testdata.516403 2026-02-22T20:56:22.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj245 testdata.516403 2026-02-22T20:56:22.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj246 testdata.516403 2026-02-22T20:56:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj247 testdata.516403 2026-02-22T20:56:22.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj248 testdata.516403 2026-02-22T20:56:22.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj249 testdata.516403 2026-02-22T20:56:22.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj250 testdata.516403 2026-02-22T20:56:22.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj251 testdata.516403 2026-02-22T20:56:22.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj252 testdata.516403 2026-02-22T20:56:22.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj253 testdata.516403 2026-02-22T20:56:22.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj254 testdata.516403 2026-02-22T20:56:22.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:22.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj255 testdata.516403 2026-02-22T20:56:23.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj256 testdata.516403 2026-02-22T20:56:23.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj257 testdata.516403 2026-02-22T20:56:23.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj258 testdata.516403 2026-02-22T20:56:23.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj259 testdata.516403 2026-02-22T20:56:23.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj260 testdata.516403 2026-02-22T20:56:23.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj261 testdata.516403 2026-02-22T20:56:23.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj262 testdata.516403 2026-02-22T20:56:23.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj263 testdata.516403 2026-02-22T20:56:23.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj264 testdata.516403 2026-02-22T20:56:23.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj265 testdata.516403 2026-02-22T20:56:23.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj266 testdata.516403 2026-02-22T20:56:23.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj267 testdata.516403 2026-02-22T20:56:23.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj268 testdata.516403 2026-02-22T20:56:23.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj269 testdata.516403 2026-02-22T20:56:23.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj270 testdata.516403 2026-02-22T20:56:23.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj271 testdata.516403 2026-02-22T20:56:23.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj272 testdata.516403 2026-02-22T20:56:23.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj273 testdata.516403 2026-02-22T20:56:23.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj274 testdata.516403 2026-02-22T20:56:23.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj275 testdata.516403 2026-02-22T20:56:23.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj276 testdata.516403 2026-02-22T20:56:23.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj277 testdata.516403 2026-02-22T20:56:23.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj278 testdata.516403 2026-02-22T20:56:23.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj279 testdata.516403 2026-02-22T20:56:23.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj280 testdata.516403 2026-02-22T20:56:23.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj281 testdata.516403 2026-02-22T20:56:23.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj282 testdata.516403 2026-02-22T20:56:23.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj283 testdata.516403 2026-02-22T20:56:23.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj284 testdata.516403 2026-02-22T20:56:23.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj285 testdata.516403 2026-02-22T20:56:23.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj286 testdata.516403 2026-02-22T20:56:23.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj287 testdata.516403 2026-02-22T20:56:23.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj288 testdata.516403 2026-02-22T20:56:23.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj289 testdata.516403 2026-02-22T20:56:23.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj290 testdata.516403 2026-02-22T20:56:23.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj291 testdata.516403 2026-02-22T20:56:23.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj292 testdata.516403 2026-02-22T20:56:23.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj293 testdata.516403 2026-02-22T20:56:23.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj294 testdata.516403 2026-02-22T20:56:23.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj295 testdata.516403 2026-02-22T20:56:23.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj296 testdata.516403 2026-02-22T20:56:23.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj297 testdata.516403 2026-02-22T20:56:23.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj298 testdata.516403 2026-02-22T20:56:23.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj299 testdata.516403 2026-02-22T20:56:23.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:23.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj300 testdata.516403 2026-02-22T20:56:24.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj301 testdata.516403 2026-02-22T20:56:24.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj302 testdata.516403 2026-02-22T20:56:24.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj303 testdata.516403 2026-02-22T20:56:24.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj304 testdata.516403 2026-02-22T20:56:24.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj305 testdata.516403 2026-02-22T20:56:24.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj306 testdata.516403 2026-02-22T20:56:24.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj307 testdata.516403 2026-02-22T20:56:24.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj308 testdata.516403 2026-02-22T20:56:24.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj309 testdata.516403 2026-02-22T20:56:24.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj310 testdata.516403 2026-02-22T20:56:24.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj311 testdata.516403 2026-02-22T20:56:24.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj312 testdata.516403 2026-02-22T20:56:24.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj313 testdata.516403 2026-02-22T20:56:24.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj314 testdata.516403 2026-02-22T20:56:24.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj315 testdata.516403 2026-02-22T20:56:24.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj316 testdata.516403 2026-02-22T20:56:24.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj317 testdata.516403 2026-02-22T20:56:24.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj318 testdata.516403 2026-02-22T20:56:24.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj319 testdata.516403 2026-02-22T20:56:24.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj320 testdata.516403 2026-02-22T20:56:24.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj321 testdata.516403 2026-02-22T20:56:24.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj322 testdata.516403 2026-02-22T20:56:24.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj323 testdata.516403 2026-02-22T20:56:24.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj324 testdata.516403 2026-02-22T20:56:24.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj325 testdata.516403 2026-02-22T20:56:24.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj326 testdata.516403 2026-02-22T20:56:24.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj327 testdata.516403 2026-02-22T20:56:24.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj328 testdata.516403 2026-02-22T20:56:24.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj329 testdata.516403 2026-02-22T20:56:24.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj330 testdata.516403 2026-02-22T20:56:24.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj331 testdata.516403 2026-02-22T20:56:24.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj332 testdata.516403 2026-02-22T20:56:24.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj333 testdata.516403 2026-02-22T20:56:24.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj334 testdata.516403 2026-02-22T20:56:24.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj335 testdata.516403 2026-02-22T20:56:24.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj336 testdata.516403 2026-02-22T20:56:24.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj337 testdata.516403 2026-02-22T20:56:24.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj338 testdata.516403 2026-02-22T20:56:24.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj339 testdata.516403 2026-02-22T20:56:24.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj340 testdata.516403 2026-02-22T20:56:24.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj341 testdata.516403 2026-02-22T20:56:24.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj342 testdata.516403 2026-02-22T20:56:24.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:24.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj343 testdata.516403 2026-02-22T20:56:25.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj344 testdata.516403 2026-02-22T20:56:25.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj345 testdata.516403 2026-02-22T20:56:25.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj346 testdata.516403 2026-02-22T20:56:25.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj347 testdata.516403 2026-02-22T20:56:25.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj348 testdata.516403 2026-02-22T20:56:25.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj349 testdata.516403 2026-02-22T20:56:25.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj350 testdata.516403 2026-02-22T20:56:25.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj351 testdata.516403 2026-02-22T20:56:25.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj352 testdata.516403 2026-02-22T20:56:25.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj353 testdata.516403 2026-02-22T20:56:25.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj354 testdata.516403 2026-02-22T20:56:25.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj355 testdata.516403 2026-02-22T20:56:25.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj356 testdata.516403 2026-02-22T20:56:25.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj357 testdata.516403 2026-02-22T20:56:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj358 testdata.516403 2026-02-22T20:56:25.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj359 testdata.516403 2026-02-22T20:56:25.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj360 testdata.516403 2026-02-22T20:56:25.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj361 testdata.516403 2026-02-22T20:56:25.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj362 testdata.516403 2026-02-22T20:56:25.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj363 testdata.516403 2026-02-22T20:56:25.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj364 testdata.516403 2026-02-22T20:56:25.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj365 testdata.516403 2026-02-22T20:56:25.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj366 testdata.516403 2026-02-22T20:56:25.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj367 testdata.516403 2026-02-22T20:56:25.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj368 testdata.516403 2026-02-22T20:56:25.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj369 testdata.516403 2026-02-22T20:56:25.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj370 testdata.516403 2026-02-22T20:56:25.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj371 testdata.516403 2026-02-22T20:56:25.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj372 testdata.516403 2026-02-22T20:56:25.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj373 testdata.516403 2026-02-22T20:56:25.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj374 testdata.516403 2026-02-22T20:56:25.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj375 testdata.516403 2026-02-22T20:56:25.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj376 testdata.516403 2026-02-22T20:56:25.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj377 testdata.516403 2026-02-22T20:56:25.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj378 testdata.516403 2026-02-22T20:56:25.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj379 testdata.516403 2026-02-22T20:56:25.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj380 testdata.516403 2026-02-22T20:56:25.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj381 testdata.516403 2026-02-22T20:56:25.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj382 testdata.516403 2026-02-22T20:56:25.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj383 testdata.516403 2026-02-22T20:56:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj384 testdata.516403 2026-02-22T20:56:25.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:25.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj385 testdata.516403 2026-02-22T20:56:26.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj386 testdata.516403 2026-02-22T20:56:26.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj387 testdata.516403 2026-02-22T20:56:26.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj388 testdata.516403 2026-02-22T20:56:26.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj389 testdata.516403 2026-02-22T20:56:26.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj390 testdata.516403 2026-02-22T20:56:26.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj391 testdata.516403 2026-02-22T20:56:26.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj392 testdata.516403 2026-02-22T20:56:26.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj393 testdata.516403 2026-02-22T20:56:26.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj394 testdata.516403 2026-02-22T20:56:26.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj395 testdata.516403 2026-02-22T20:56:26.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj396 testdata.516403 2026-02-22T20:56:26.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj397 testdata.516403 2026-02-22T20:56:26.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj398 testdata.516403 2026-02-22T20:56:26.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj399 testdata.516403 2026-02-22T20:56:26.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj400 testdata.516403 2026-02-22T20:56:26.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj401 testdata.516403 2026-02-22T20:56:26.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj402 testdata.516403 2026-02-22T20:56:26.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj403 testdata.516403 2026-02-22T20:56:26.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj404 testdata.516403 2026-02-22T20:56:26.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj405 testdata.516403 2026-02-22T20:56:26.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj406 testdata.516403 2026-02-22T20:56:26.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj407 testdata.516403 2026-02-22T20:56:26.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj408 testdata.516403 2026-02-22T20:56:26.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj409 testdata.516403 2026-02-22T20:56:26.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj410 testdata.516403 2026-02-22T20:56:26.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj411 testdata.516403 2026-02-22T20:56:26.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj412 testdata.516403 2026-02-22T20:56:26.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj413 testdata.516403 2026-02-22T20:56:26.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj414 testdata.516403 2026-02-22T20:56:26.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj415 testdata.516403 2026-02-22T20:56:26.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj416 testdata.516403 2026-02-22T20:56:26.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj417 testdata.516403 2026-02-22T20:56:26.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj418 testdata.516403 2026-02-22T20:56:26.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj419 testdata.516403 2026-02-22T20:56:26.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj420 testdata.516403 2026-02-22T20:56:26.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj421 testdata.516403 2026-02-22T20:56:26.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj422 testdata.516403 2026-02-22T20:56:26.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj423 testdata.516403 2026-02-22T20:56:26.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj424 testdata.516403 2026-02-22T20:56:26.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj425 testdata.516403 2026-02-22T20:56:26.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj426 testdata.516403 2026-02-22T20:56:26.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj427 testdata.516403 2026-02-22T20:56:26.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj428 testdata.516403 2026-02-22T20:56:26.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:26.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj429 testdata.516403 2026-02-22T20:56:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj430 testdata.516403 2026-02-22T20:56:27.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj431 testdata.516403 2026-02-22T20:56:27.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj432 testdata.516403 2026-02-22T20:56:27.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj433 testdata.516403 2026-02-22T20:56:27.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj434 testdata.516403 2026-02-22T20:56:27.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj435 testdata.516403 2026-02-22T20:56:27.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj436 testdata.516403 2026-02-22T20:56:27.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj437 testdata.516403 2026-02-22T20:56:27.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj438 testdata.516403 2026-02-22T20:56:27.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj439 testdata.516403 2026-02-22T20:56:27.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj440 testdata.516403 2026-02-22T20:56:27.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj441 testdata.516403 2026-02-22T20:56:27.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj442 testdata.516403 2026-02-22T20:56:27.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj443 testdata.516403 2026-02-22T20:56:27.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj444 testdata.516403 2026-02-22T20:56:27.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj445 testdata.516403 2026-02-22T20:56:27.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj446 testdata.516403 2026-02-22T20:56:27.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj447 testdata.516403 2026-02-22T20:56:27.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj448 testdata.516403 2026-02-22T20:56:27.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj449 testdata.516403 2026-02-22T20:56:27.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj450 testdata.516403 2026-02-22T20:56:27.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj451 testdata.516403 2026-02-22T20:56:27.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj452 testdata.516403 2026-02-22T20:56:27.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj453 testdata.516403 2026-02-22T20:56:27.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj454 testdata.516403 2026-02-22T20:56:27.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj455 testdata.516403 2026-02-22T20:56:27.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj456 testdata.516403 2026-02-22T20:56:27.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj457 testdata.516403 2026-02-22T20:56:27.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj458 testdata.516403 2026-02-22T20:56:27.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj459 testdata.516403 2026-02-22T20:56:27.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj460 testdata.516403 2026-02-22T20:56:27.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj461 testdata.516403 2026-02-22T20:56:27.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj462 testdata.516403 2026-02-22T20:56:27.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj463 testdata.516403 2026-02-22T20:56:27.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj464 testdata.516403 2026-02-22T20:56:27.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj465 testdata.516403 2026-02-22T20:56:27.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj466 testdata.516403 2026-02-22T20:56:27.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj467 testdata.516403 2026-02-22T20:56:27.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj468 testdata.516403 2026-02-22T20:56:27.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj469 testdata.516403 2026-02-22T20:56:27.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj470 testdata.516403 2026-02-22T20:56:27.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj471 testdata.516403 2026-02-22T20:56:27.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:27.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj472 testdata.516403 2026-02-22T20:56:28.018 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj473 testdata.516403 2026-02-22T20:56:28.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj474 testdata.516403 2026-02-22T20:56:28.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj475 testdata.516403 2026-02-22T20:56:28.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj476 testdata.516403 2026-02-22T20:56:28.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj477 testdata.516403 2026-02-22T20:56:28.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj478 testdata.516403 2026-02-22T20:56:28.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj479 testdata.516403 2026-02-22T20:56:28.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj480 testdata.516403 2026-02-22T20:56:28.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj481 testdata.516403 2026-02-22T20:56:28.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj482 testdata.516403 2026-02-22T20:56:28.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj483 testdata.516403 2026-02-22T20:56:28.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj484 testdata.516403 2026-02-22T20:56:28.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj485 testdata.516403 2026-02-22T20:56:28.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj486 testdata.516403 2026-02-22T20:56:28.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj487 testdata.516403 2026-02-22T20:56:28.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj488 testdata.516403 2026-02-22T20:56:28.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj489 testdata.516403 2026-02-22T20:56:28.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj490 testdata.516403 2026-02-22T20:56:28.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj491 testdata.516403 2026-02-22T20:56:28.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj492 testdata.516403 2026-02-22T20:56:28.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj493 testdata.516403 2026-02-22T20:56:28.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj494 testdata.516403 2026-02-22T20:56:28.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj495 testdata.516403 2026-02-22T20:56:28.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj496 testdata.516403 2026-02-22T20:56:28.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj497 testdata.516403 2026-02-22T20:56:28.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj498 testdata.516403 2026-02-22T20:56:28.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj499 testdata.516403 2026-02-22T20:56:28.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj500 testdata.516403 2026-02-22T20:56:28.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj501 testdata.516403 2026-02-22T20:56:28.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj502 testdata.516403 2026-02-22T20:56:28.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj503 testdata.516403 2026-02-22T20:56:28.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj504 testdata.516403 2026-02-22T20:56:28.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj505 testdata.516403 2026-02-22T20:56:28.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj506 testdata.516403 2026-02-22T20:56:28.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj507 testdata.516403 2026-02-22T20:56:28.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj508 testdata.516403 2026-02-22T20:56:28.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj509 testdata.516403 2026-02-22T20:56:28.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj510 testdata.516403 2026-02-22T20:56:28.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj511 testdata.516403 2026-02-22T20:56:28.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj512 testdata.516403 2026-02-22T20:56:28.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj513 testdata.516403 2026-02-22T20:56:28.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:28.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj514 testdata.516403 2026-02-22T20:56:29.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj515 testdata.516403 2026-02-22T20:56:29.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj516 testdata.516403 2026-02-22T20:56:29.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj517 testdata.516403 2026-02-22T20:56:29.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj518 testdata.516403 2026-02-22T20:56:29.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj519 testdata.516403 2026-02-22T20:56:29.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj520 testdata.516403 2026-02-22T20:56:29.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj521 testdata.516403 2026-02-22T20:56:29.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj522 testdata.516403 2026-02-22T20:56:29.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj523 testdata.516403 2026-02-22T20:56:29.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj524 testdata.516403 2026-02-22T20:56:29.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj525 testdata.516403 2026-02-22T20:56:29.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj526 testdata.516403 2026-02-22T20:56:29.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj527 testdata.516403 2026-02-22T20:56:29.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj528 testdata.516403 2026-02-22T20:56:29.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj529 testdata.516403 2026-02-22T20:56:29.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj530 testdata.516403 2026-02-22T20:56:29.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj531 testdata.516403 2026-02-22T20:56:29.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj532 testdata.516403 2026-02-22T20:56:29.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj533 testdata.516403 2026-02-22T20:56:29.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj534 testdata.516403 2026-02-22T20:56:29.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj535 testdata.516403 2026-02-22T20:56:29.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj536 testdata.516403 2026-02-22T20:56:29.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj537 testdata.516403 2026-02-22T20:56:29.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj538 testdata.516403 2026-02-22T20:56:29.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj539 testdata.516403 2026-02-22T20:56:29.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj540 testdata.516403 2026-02-22T20:56:29.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj541 testdata.516403 2026-02-22T20:56:29.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj542 testdata.516403 2026-02-22T20:56:29.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj543 testdata.516403 2026-02-22T20:56:29.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj544 testdata.516403 2026-02-22T20:56:29.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj545 testdata.516403 2026-02-22T20:56:29.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj546 testdata.516403 2026-02-22T20:56:29.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj547 testdata.516403 2026-02-22T20:56:29.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj548 testdata.516403 2026-02-22T20:56:29.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj549 testdata.516403 2026-02-22T20:56:29.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj550 testdata.516403 2026-02-22T20:56:29.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj551 testdata.516403 2026-02-22T20:56:29.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj552 testdata.516403 2026-02-22T20:56:29.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj553 testdata.516403 2026-02-22T20:56:29.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj554 testdata.516403 2026-02-22T20:56:29.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj555 testdata.516403 2026-02-22T20:56:29.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj556 testdata.516403 2026-02-22T20:56:29.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj557 testdata.516403 2026-02-22T20:56:29.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:29.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj558 testdata.516403 2026-02-22T20:56:30.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj559 testdata.516403 2026-02-22T20:56:30.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj560 testdata.516403 2026-02-22T20:56:30.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj561 testdata.516403 2026-02-22T20:56:30.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj562 testdata.516403 2026-02-22T20:56:30.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj563 testdata.516403 2026-02-22T20:56:30.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj564 testdata.516403 2026-02-22T20:56:30.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj565 testdata.516403 2026-02-22T20:56:30.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj566 testdata.516403 2026-02-22T20:56:30.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj567 testdata.516403 2026-02-22T20:56:30.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj568 testdata.516403 2026-02-22T20:56:30.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj569 testdata.516403 2026-02-22T20:56:30.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj570 testdata.516403 2026-02-22T20:56:30.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj571 testdata.516403 2026-02-22T20:56:30.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj572 testdata.516403 2026-02-22T20:56:30.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj573 testdata.516403 2026-02-22T20:56:30.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj574 testdata.516403 2026-02-22T20:56:30.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj575 testdata.516403 2026-02-22T20:56:30.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj576 testdata.516403 2026-02-22T20:56:30.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj577 testdata.516403 2026-02-22T20:56:30.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj578 testdata.516403 2026-02-22T20:56:30.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj579 testdata.516403 2026-02-22T20:56:30.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj580 testdata.516403 2026-02-22T20:56:30.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj581 testdata.516403 2026-02-22T20:56:30.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj582 testdata.516403 2026-02-22T20:56:30.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj583 testdata.516403 2026-02-22T20:56:30.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj584 testdata.516403 2026-02-22T20:56:30.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj585 testdata.516403 2026-02-22T20:56:30.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj586 testdata.516403 2026-02-22T20:56:30.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj587 testdata.516403 2026-02-22T20:56:30.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj588 testdata.516403 2026-02-22T20:56:30.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj589 testdata.516403 2026-02-22T20:56:30.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj590 testdata.516403 2026-02-22T20:56:30.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj591 testdata.516403 2026-02-22T20:56:30.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj592 testdata.516403 2026-02-22T20:56:30.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj593 testdata.516403 2026-02-22T20:56:30.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj594 testdata.516403 2026-02-22T20:56:30.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj595 testdata.516403 2026-02-22T20:56:30.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj596 testdata.516403 2026-02-22T20:56:30.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj597 testdata.516403 2026-02-22T20:56:30.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj598 testdata.516403 2026-02-22T20:56:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj599 testdata.516403 2026-02-22T20:56:30.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj600 testdata.516403 2026-02-22T20:56:30.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:30.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj601 testdata.516403 2026-02-22T20:56:31.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj602 testdata.516403 2026-02-22T20:56:31.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj603 testdata.516403 2026-02-22T20:56:31.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj604 testdata.516403 2026-02-22T20:56:31.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj605 testdata.516403 2026-02-22T20:56:31.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj606 testdata.516403 2026-02-22T20:56:31.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj607 testdata.516403 2026-02-22T20:56:31.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj608 testdata.516403 2026-02-22T20:56:31.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj609 testdata.516403 2026-02-22T20:56:31.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj610 testdata.516403 2026-02-22T20:56:31.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj611 testdata.516403 2026-02-22T20:56:31.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj612 testdata.516403 2026-02-22T20:56:31.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj613 testdata.516403 2026-02-22T20:56:31.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj614 testdata.516403 2026-02-22T20:56:31.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj615 testdata.516403 2026-02-22T20:56:31.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj616 testdata.516403 2026-02-22T20:56:31.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj617 testdata.516403 2026-02-22T20:56:31.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj618 testdata.516403 2026-02-22T20:56:31.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj619 testdata.516403 2026-02-22T20:56:31.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj620 testdata.516403 2026-02-22T20:56:31.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj621 testdata.516403 2026-02-22T20:56:31.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj622 testdata.516403 2026-02-22T20:56:31.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj623 testdata.516403 2026-02-22T20:56:31.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj624 testdata.516403 2026-02-22T20:56:31.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj625 testdata.516403 2026-02-22T20:56:31.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj626 testdata.516403 2026-02-22T20:56:31.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj627 testdata.516403 2026-02-22T20:56:31.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj628 testdata.516403 2026-02-22T20:56:31.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj629 testdata.516403 2026-02-22T20:56:31.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj630 testdata.516403 2026-02-22T20:56:31.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj631 testdata.516403 2026-02-22T20:56:31.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj632 testdata.516403 2026-02-22T20:56:31.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj633 testdata.516403 2026-02-22T20:56:31.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj634 testdata.516403 2026-02-22T20:56:31.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj635 testdata.516403 2026-02-22T20:56:31.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj636 testdata.516403 2026-02-22T20:56:31.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj637 testdata.516403 2026-02-22T20:56:31.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj638 testdata.516403 2026-02-22T20:56:31.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj639 testdata.516403 2026-02-22T20:56:31.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj640 testdata.516403 2026-02-22T20:56:31.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj641 testdata.516403 2026-02-22T20:56:31.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj642 testdata.516403 2026-02-22T20:56:31.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:31.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj643 testdata.516403 2026-02-22T20:56:32.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj644 testdata.516403 2026-02-22T20:56:32.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj645 testdata.516403 2026-02-22T20:56:32.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj646 testdata.516403 2026-02-22T20:56:32.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj647 testdata.516403 2026-02-22T20:56:32.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj648 testdata.516403 2026-02-22T20:56:32.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj649 testdata.516403 2026-02-22T20:56:32.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj650 testdata.516403 2026-02-22T20:56:32.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj651 testdata.516403 2026-02-22T20:56:32.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj652 testdata.516403 2026-02-22T20:56:32.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj653 testdata.516403 2026-02-22T20:56:32.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj654 testdata.516403 2026-02-22T20:56:32.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj655 testdata.516403 2026-02-22T20:56:32.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj656 testdata.516403 2026-02-22T20:56:32.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj657 testdata.516403 2026-02-22T20:56:32.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj658 testdata.516403 2026-02-22T20:56:32.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj659 testdata.516403 2026-02-22T20:56:32.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj660 testdata.516403 2026-02-22T20:56:32.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj661 testdata.516403 2026-02-22T20:56:32.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj662 testdata.516403 2026-02-22T20:56:32.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj663 testdata.516403 2026-02-22T20:56:32.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj664 testdata.516403 2026-02-22T20:56:32.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj665 testdata.516403 2026-02-22T20:56:32.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj666 testdata.516403 2026-02-22T20:56:32.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj667 testdata.516403 2026-02-22T20:56:32.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj668 testdata.516403 2026-02-22T20:56:32.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj669 testdata.516403 2026-02-22T20:56:32.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj670 testdata.516403 2026-02-22T20:56:32.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj671 testdata.516403 2026-02-22T20:56:32.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj672 testdata.516403 2026-02-22T20:56:32.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj673 testdata.516403 2026-02-22T20:56:32.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj674 testdata.516403 2026-02-22T20:56:32.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj675 testdata.516403 2026-02-22T20:56:32.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj676 testdata.516403 2026-02-22T20:56:32.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj677 testdata.516403 2026-02-22T20:56:32.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj678 testdata.516403 2026-02-22T20:56:32.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj679 testdata.516403 2026-02-22T20:56:32.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj680 testdata.516403 2026-02-22T20:56:32.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj681 testdata.516403 2026-02-22T20:56:32.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj682 testdata.516403 2026-02-22T20:56:32.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj683 testdata.516403 2026-02-22T20:56:32.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj684 testdata.516403 2026-02-22T20:56:32.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj685 testdata.516403 2026-02-22T20:56:32.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj686 testdata.516403 2026-02-22T20:56:32.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj687 testdata.516403 2026-02-22T20:56:32.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:32.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj688 testdata.516403 2026-02-22T20:56:33.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj689 testdata.516403 2026-02-22T20:56:33.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj690 testdata.516403 2026-02-22T20:56:33.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj691 testdata.516403 2026-02-22T20:56:33.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj692 testdata.516403 2026-02-22T20:56:33.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj693 testdata.516403 2026-02-22T20:56:33.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj694 testdata.516403 2026-02-22T20:56:33.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj695 testdata.516403 2026-02-22T20:56:33.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj696 testdata.516403 2026-02-22T20:56:33.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj697 testdata.516403 2026-02-22T20:56:33.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj698 testdata.516403 2026-02-22T20:56:33.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj699 testdata.516403 2026-02-22T20:56:33.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj700 testdata.516403 2026-02-22T20:56:33.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj701 testdata.516403 2026-02-22T20:56:33.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj702 testdata.516403 2026-02-22T20:56:33.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj703 testdata.516403 2026-02-22T20:56:33.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj704 testdata.516403 2026-02-22T20:56:33.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj705 testdata.516403 2026-02-22T20:56:33.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj706 testdata.516403 2026-02-22T20:56:33.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj707 testdata.516403 2026-02-22T20:56:33.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj708 testdata.516403 2026-02-22T20:56:33.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj709 testdata.516403 2026-02-22T20:56:33.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj710 testdata.516403 2026-02-22T20:56:33.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj711 testdata.516403 2026-02-22T20:56:33.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj712 testdata.516403 2026-02-22T20:56:33.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj713 testdata.516403 2026-02-22T20:56:33.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj714 testdata.516403 2026-02-22T20:56:33.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj715 testdata.516403 2026-02-22T20:56:33.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj716 testdata.516403 2026-02-22T20:56:33.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj717 testdata.516403 2026-02-22T20:56:33.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj718 testdata.516403 2026-02-22T20:56:33.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj719 testdata.516403 2026-02-22T20:56:33.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj720 testdata.516403 2026-02-22T20:56:33.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj721 testdata.516403 2026-02-22T20:56:33.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj722 testdata.516403 2026-02-22T20:56:33.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj723 testdata.516403 2026-02-22T20:56:33.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj724 testdata.516403 2026-02-22T20:56:33.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj725 testdata.516403 2026-02-22T20:56:33.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj726 testdata.516403 2026-02-22T20:56:33.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj727 testdata.516403 2026-02-22T20:56:33.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj728 testdata.516403 2026-02-22T20:56:33.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj729 testdata.516403 2026-02-22T20:56:33.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj730 testdata.516403 2026-02-22T20:56:33.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:33.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj731 testdata.516403 2026-02-22T20:56:34.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj732 testdata.516403 2026-02-22T20:56:34.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj733 testdata.516403 2026-02-22T20:56:34.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj734 testdata.516403 2026-02-22T20:56:34.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj735 testdata.516403 2026-02-22T20:56:34.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj736 testdata.516403 2026-02-22T20:56:34.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj737 testdata.516403 2026-02-22T20:56:34.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj738 testdata.516403 2026-02-22T20:56:34.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj739 testdata.516403 2026-02-22T20:56:34.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj740 testdata.516403 2026-02-22T20:56:34.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj741 testdata.516403 2026-02-22T20:56:34.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj742 testdata.516403 2026-02-22T20:56:34.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj743 testdata.516403 2026-02-22T20:56:34.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj744 testdata.516403 2026-02-22T20:56:34.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj745 testdata.516403 2026-02-22T20:56:34.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj746 testdata.516403 2026-02-22T20:56:34.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj747 testdata.516403 2026-02-22T20:56:34.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj748 testdata.516403 2026-02-22T20:56:34.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj749 testdata.516403 2026-02-22T20:56:34.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj750 testdata.516403 2026-02-22T20:56:34.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj751 testdata.516403 2026-02-22T20:56:34.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj752 testdata.516403 2026-02-22T20:56:34.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj753 testdata.516403 2026-02-22T20:56:34.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj754 testdata.516403 2026-02-22T20:56:34.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj755 testdata.516403 2026-02-22T20:56:34.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj756 testdata.516403 2026-02-22T20:56:34.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj757 testdata.516403 2026-02-22T20:56:34.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj758 testdata.516403 2026-02-22T20:56:34.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj759 testdata.516403 2026-02-22T20:56:34.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj760 testdata.516403 2026-02-22T20:56:34.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj761 testdata.516403 2026-02-22T20:56:34.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj762 testdata.516403 2026-02-22T20:56:34.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj763 testdata.516403 2026-02-22T20:56:34.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj764 testdata.516403 2026-02-22T20:56:34.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj765 testdata.516403 2026-02-22T20:56:34.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj766 testdata.516403 2026-02-22T20:56:34.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj767 testdata.516403 2026-02-22T20:56:34.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj768 testdata.516403 2026-02-22T20:56:34.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj769 testdata.516403 2026-02-22T20:56:34.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj770 testdata.516403 2026-02-22T20:56:34.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj771 testdata.516403 2026-02-22T20:56:34.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj772 testdata.516403 2026-02-22T20:56:34.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj773 testdata.516403 2026-02-22T20:56:34.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj774 testdata.516403 2026-02-22T20:56:34.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:34.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj775 testdata.516403 2026-02-22T20:56:35.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj776 testdata.516403 2026-02-22T20:56:35.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj777 testdata.516403 2026-02-22T20:56:35.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj778 testdata.516403 2026-02-22T20:56:35.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj779 testdata.516403 2026-02-22T20:56:35.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj780 testdata.516403 2026-02-22T20:56:35.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj781 testdata.516403 2026-02-22T20:56:35.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj782 testdata.516403 2026-02-22T20:56:35.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj783 testdata.516403 2026-02-22T20:56:35.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj784 testdata.516403 2026-02-22T20:56:35.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj785 testdata.516403 2026-02-22T20:56:35.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj786 testdata.516403 2026-02-22T20:56:35.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj787 testdata.516403 2026-02-22T20:56:35.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj788 testdata.516403 2026-02-22T20:56:35.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj789 testdata.516403 2026-02-22T20:56:35.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj790 testdata.516403 2026-02-22T20:56:35.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj791 testdata.516403 2026-02-22T20:56:35.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj792 testdata.516403 2026-02-22T20:56:35.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj793 testdata.516403 2026-02-22T20:56:35.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj794 testdata.516403 2026-02-22T20:56:35.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.466 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj795 testdata.516403 2026-02-22T20:56:35.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj796 testdata.516403 2026-02-22T20:56:35.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj797 testdata.516403 2026-02-22T20:56:35.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj798 testdata.516403 2026-02-22T20:56:35.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj799 testdata.516403 2026-02-22T20:56:35.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj800 testdata.516403 2026-02-22T20:56:35.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj801 testdata.516403 2026-02-22T20:56:35.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj802 testdata.516403 2026-02-22T20:56:35.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj803 testdata.516403 2026-02-22T20:56:35.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj804 testdata.516403 2026-02-22T20:56:35.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj805 testdata.516403 2026-02-22T20:56:35.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj806 testdata.516403 2026-02-22T20:56:35.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj807 testdata.516403 2026-02-22T20:56:35.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj808 testdata.516403 2026-02-22T20:56:35.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj809 testdata.516403 2026-02-22T20:56:35.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj810 testdata.516403 2026-02-22T20:56:35.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj811 testdata.516403 2026-02-22T20:56:35.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj812 testdata.516403 2026-02-22T20:56:35.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj813 testdata.516403 2026-02-22T20:56:35.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj814 testdata.516403 2026-02-22T20:56:35.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj815 testdata.516403 2026-02-22T20:56:35.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj816 testdata.516403 2026-02-22T20:56:35.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:35.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj817 testdata.516403 2026-02-22T20:56:36.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj818 testdata.516403 2026-02-22T20:56:36.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj819 testdata.516403 2026-02-22T20:56:36.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj820 testdata.516403 2026-02-22T20:56:36.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj821 testdata.516403 2026-02-22T20:56:36.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj822 testdata.516403 2026-02-22T20:56:36.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj823 testdata.516403 2026-02-22T20:56:36.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj824 testdata.516403 2026-02-22T20:56:36.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj825 testdata.516403 2026-02-22T20:56:36.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj826 testdata.516403 2026-02-22T20:56:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj827 testdata.516403 2026-02-22T20:56:36.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj828 testdata.516403 2026-02-22T20:56:36.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj829 testdata.516403 2026-02-22T20:56:36.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj830 testdata.516403 2026-02-22T20:56:36.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj831 testdata.516403 2026-02-22T20:56:36.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj832 testdata.516403 2026-02-22T20:56:36.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj833 testdata.516403 2026-02-22T20:56:36.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj834 testdata.516403 2026-02-22T20:56:36.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj835 testdata.516403 2026-02-22T20:56:36.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj836 testdata.516403 2026-02-22T20:56:36.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj837 testdata.516403 2026-02-22T20:56:36.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj838 testdata.516403 2026-02-22T20:56:36.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj839 testdata.516403 2026-02-22T20:56:36.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj840 testdata.516403 2026-02-22T20:56:36.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj841 testdata.516403 2026-02-22T20:56:36.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj842 testdata.516403 2026-02-22T20:56:36.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj843 testdata.516403 2026-02-22T20:56:36.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj844 testdata.516403 2026-02-22T20:56:36.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj845 testdata.516403 2026-02-22T20:56:36.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj846 testdata.516403 2026-02-22T20:56:36.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj847 testdata.516403 2026-02-22T20:56:36.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj848 testdata.516403 2026-02-22T20:56:36.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj849 testdata.516403 2026-02-22T20:56:36.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj850 testdata.516403 2026-02-22T20:56:36.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj851 testdata.516403 2026-02-22T20:56:36.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj852 testdata.516403 2026-02-22T20:56:36.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj853 testdata.516403 2026-02-22T20:56:36.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj854 testdata.516403 2026-02-22T20:56:36.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj855 testdata.516403 2026-02-22T20:56:36.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj856 testdata.516403 2026-02-22T20:56:36.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj857 testdata.516403 2026-02-22T20:56:36.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj858 testdata.516403 2026-02-22T20:56:36.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:36.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj859 testdata.516403 2026-02-22T20:56:37.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj860 testdata.516403 2026-02-22T20:56:37.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj861 testdata.516403 2026-02-22T20:56:37.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj862 testdata.516403 2026-02-22T20:56:37.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj863 testdata.516403 2026-02-22T20:56:37.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj864 testdata.516403 2026-02-22T20:56:37.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj865 testdata.516403 2026-02-22T20:56:37.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj866 testdata.516403 2026-02-22T20:56:37.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj867 testdata.516403 2026-02-22T20:56:37.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj868 testdata.516403 2026-02-22T20:56:37.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj869 testdata.516403 2026-02-22T20:56:37.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj870 testdata.516403 2026-02-22T20:56:37.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj871 testdata.516403 2026-02-22T20:56:37.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj872 testdata.516403 2026-02-22T20:56:37.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj873 testdata.516403 2026-02-22T20:56:37.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj874 testdata.516403 2026-02-22T20:56:37.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj875 testdata.516403 2026-02-22T20:56:37.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj876 testdata.516403 2026-02-22T20:56:37.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj877 testdata.516403 2026-02-22T20:56:37.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj878 testdata.516403 2026-02-22T20:56:37.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj879 testdata.516403 2026-02-22T20:56:37.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj880 testdata.516403 2026-02-22T20:56:37.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj881 testdata.516403 2026-02-22T20:56:37.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj882 testdata.516403 2026-02-22T20:56:37.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj883 testdata.516403 2026-02-22T20:56:37.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj884 testdata.516403 2026-02-22T20:56:37.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj885 testdata.516403 2026-02-22T20:56:37.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj886 testdata.516403 2026-02-22T20:56:37.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj887 testdata.516403 2026-02-22T20:56:37.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj888 testdata.516403 2026-02-22T20:56:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj889 testdata.516403 2026-02-22T20:56:37.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj890 testdata.516403 2026-02-22T20:56:37.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj891 testdata.516403 2026-02-22T20:56:37.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj892 testdata.516403 2026-02-22T20:56:37.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj893 testdata.516403 2026-02-22T20:56:37.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj894 testdata.516403 2026-02-22T20:56:37.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj895 testdata.516403 2026-02-22T20:56:37.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj896 testdata.516403 2026-02-22T20:56:37.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj897 testdata.516403 2026-02-22T20:56:37.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj898 testdata.516403 2026-02-22T20:56:37.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj899 testdata.516403 2026-02-22T20:56:37.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj900 testdata.516403 2026-02-22T20:56:37.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:37.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj901 testdata.516403 2026-02-22T20:56:38.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj902 testdata.516403 2026-02-22T20:56:38.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj903 testdata.516403 2026-02-22T20:56:38.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj904 testdata.516403 2026-02-22T20:56:38.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj905 testdata.516403 2026-02-22T20:56:38.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj906 testdata.516403 2026-02-22T20:56:38.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj907 testdata.516403 2026-02-22T20:56:38.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj908 testdata.516403 2026-02-22T20:56:38.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj909 testdata.516403 2026-02-22T20:56:38.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj910 testdata.516403 2026-02-22T20:56:38.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj911 testdata.516403 2026-02-22T20:56:38.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj912 testdata.516403 2026-02-22T20:56:38.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.262 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj913 testdata.516403 2026-02-22T20:56:38.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj914 testdata.516403 2026-02-22T20:56:38.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj915 testdata.516403 2026-02-22T20:56:38.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj916 testdata.516403 2026-02-22T20:56:38.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj917 testdata.516403 2026-02-22T20:56:38.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj918 testdata.516403 2026-02-22T20:56:38.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj919 testdata.516403 2026-02-22T20:56:38.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj920 testdata.516403 2026-02-22T20:56:38.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj921 testdata.516403 2026-02-22T20:56:38.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj922 testdata.516403 2026-02-22T20:56:38.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj923 testdata.516403 2026-02-22T20:56:38.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj924 testdata.516403 2026-02-22T20:56:38.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj925 testdata.516403 2026-02-22T20:56:38.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.565 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj926 testdata.516403 2026-02-22T20:56:38.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj927 testdata.516403 2026-02-22T20:56:38.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj928 testdata.516403 2026-02-22T20:56:38.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj929 testdata.516403 2026-02-22T20:56:38.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj930 testdata.516403 2026-02-22T20:56:38.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj931 testdata.516403 2026-02-22T20:56:38.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj932 testdata.516403 2026-02-22T20:56:38.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj933 testdata.516403 2026-02-22T20:56:38.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj934 testdata.516403 2026-02-22T20:56:38.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj935 testdata.516403 2026-02-22T20:56:38.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj936 testdata.516403 2026-02-22T20:56:38.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj937 testdata.516403 2026-02-22T20:56:38.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj938 testdata.516403 2026-02-22T20:56:38.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj939 testdata.516403 2026-02-22T20:56:38.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj940 testdata.516403 2026-02-22T20:56:38.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj941 testdata.516403 2026-02-22T20:56:38.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj942 testdata.516403 2026-02-22T20:56:38.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj943 testdata.516403 2026-02-22T20:56:38.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:38.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj944 testdata.516403 2026-02-22T20:56:39.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj945 testdata.516403 2026-02-22T20:56:39.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj946 testdata.516403 2026-02-22T20:56:39.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj947 testdata.516403 2026-02-22T20:56:39.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj948 testdata.516403 2026-02-22T20:56:39.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj949 testdata.516403 2026-02-22T20:56:39.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj950 testdata.516403 2026-02-22T20:56:39.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj951 testdata.516403 2026-02-22T20:56:39.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj952 testdata.516403 2026-02-22T20:56:39.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj953 testdata.516403 2026-02-22T20:56:39.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj954 testdata.516403 2026-02-22T20:56:39.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj955 testdata.516403 2026-02-22T20:56:39.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj956 testdata.516403 2026-02-22T20:56:39.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj957 testdata.516403 2026-02-22T20:56:39.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj958 testdata.516403 2026-02-22T20:56:39.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj959 testdata.516403 2026-02-22T20:56:39.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj960 testdata.516403 2026-02-22T20:56:39.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj961 testdata.516403 2026-02-22T20:56:39.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj962 testdata.516403 2026-02-22T20:56:39.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj963 testdata.516403 2026-02-22T20:56:39.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj964 testdata.516403 2026-02-22T20:56:39.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj965 testdata.516403 2026-02-22T20:56:39.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj966 testdata.516403 2026-02-22T20:56:39.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj967 testdata.516403 2026-02-22T20:56:39.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj968 testdata.516403 2026-02-22T20:56:39.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj969 testdata.516403 2026-02-22T20:56:39.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj970 testdata.516403 2026-02-22T20:56:39.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj971 testdata.516403 2026-02-22T20:56:39.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj972 testdata.516403 2026-02-22T20:56:39.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj973 testdata.516403 2026-02-22T20:56:39.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj974 testdata.516403 2026-02-22T20:56:39.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj975 testdata.516403 2026-02-22T20:56:39.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj976 testdata.516403 2026-02-22T20:56:39.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj977 testdata.516403 2026-02-22T20:56:39.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj978 testdata.516403 2026-02-22T20:56:39.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj979 testdata.516403 2026-02-22T20:56:39.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:39.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj980 testdata.516403 2026-02-22T20:56:40.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj981 testdata.516403 2026-02-22T20:56:40.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj982 testdata.516403 2026-02-22T20:56:40.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj983 testdata.516403 2026-02-22T20:56:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj984 testdata.516403 2026-02-22T20:56:40.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj985 testdata.516403 2026-02-22T20:56:40.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj986 testdata.516403 2026-02-22T20:56:40.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj987 testdata.516403 2026-02-22T20:56:40.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj988 testdata.516403 2026-02-22T20:56:40.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj989 testdata.516403 2026-02-22T20:56:40.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj990 testdata.516403 2026-02-22T20:56:40.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj991 testdata.516403 2026-02-22T20:56:40.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj992 testdata.516403 2026-02-22T20:56:40.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj993 testdata.516403 2026-02-22T20:56:40.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj994 testdata.516403 2026-02-22T20:56:40.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj995 testdata.516403 2026-02-22T20:56:40.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj996 testdata.516403 2026-02-22T20:56:40.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj997 testdata.516403 2026-02-22T20:56:40.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj998 testdata.516403 2026-02-22T20:56:40.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj999 testdata.516403 2026-02-22T20:56:40.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T20:56:40.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj1000 testdata.516403 2026-02-22T20:56:40.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: rm -f testdata.516403 2026-02-22T20:56:40.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: get_primary test obj1 2026-02-22T20:56:40.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-22T20:56:40.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T20:56:40.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-22T20:56:40.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T20:56:40.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: local primary=1 2026-02-22T20:56:40.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: local pgid=1.0 2026-02-22T20:56:40.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: ceph tell 1.0 schedule-deep-scrub 2026-02-22T20:56:40.687 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T20:56:40.688 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T20:56:40.688 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T20:56:40.688 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T16:08:30.682666+0000" 2026-02-22T20:56:40.688 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T20:56:40.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:342: _scrub_abort: set -o pipefail 2026-02-22T20:56:40.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: found=no 2026-02-22T20:56:40.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: seq 0 200 2026-02-22T20:56:40.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:56:40.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: flush_pg_stats 2026-02-22T20:56:40.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:56:40.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:56:40.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:56:40.857 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:56:40.858 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:56:40.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:56:40.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:40.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:56:40.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836515 2026-02-22T20:56:40.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836515 2026-02-22T20:56:40.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515' 2026-02-22T20:56:40.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:40.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:56:40.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705696 2026-02-22T20:56:40.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705696 2026-02-22T20:56:40.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-38654705696' 2026-02-22T20:56:40.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:40.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:56:41.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574878 2026-02-22T20:56:41.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574878 2026-02-22T20:56:41.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-38654705696 2-55834574878' 2026-02-22T20:56:41.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:41.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836515 2026-02-22T20:56:41.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:41.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:56:41.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836515 2026-02-22T20:56:41.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:41.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836515 2026-02-22T20:56:41.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2026-02-22T20:56:41.077 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836515 2026-02-22T20:56:41.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:41.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836515 2026-02-22T20:56:41.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:56:42.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:56:42.233 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:42.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836515 2026-02-22T20:56:42.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:42.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705696 2026-02-22T20:56:42.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:42.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:56:42.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705696 2026-02-22T20:56:42.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:42.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705696 2026-02-22T20:56:42.396 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705696 2026-02-22T20:56:42.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705696' 2026-02-22T20:56:42.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:56:42.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705697 -lt 38654705696 2026-02-22T20:56:42.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:42.562 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574878 2026-02-22T20:56:42.562 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:42.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:56:42.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574878 2026-02-22T20:56:42.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:42.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574878 2026-02-22T20:56:42.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574878' 2026-02-22T20:56:42.566 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574878 2026-02-22T20:56:42.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:56:42.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574878 -lt 55834574878 2026-02-22T20:56:42.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: ceph pg dump pgs 2026-02-22T20:56:42.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep '^1.0' 2026-02-22T20:56:42.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep -q scrubbing 2026-02-22T20:56:42.880 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:56:42.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:56:42.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: flush_pg_stats 2026-02-22T20:56:42.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:56:42.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:56:43.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:56:43.045 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:56:43.045 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:56:43.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:56:43.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:43.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:56:43.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836518 2026-02-22T20:56:43.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836518 2026-02-22T20:56:43.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518' 2026-02-22T20:56:43.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:43.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:56:43.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705700 2026-02-22T20:56:43.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705700 2026-02-22T20:56:43.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705700' 2026-02-22T20:56:43.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:43.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:56:43.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574881 2026-02-22T20:56:43.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574881 2026-02-22T20:56:43.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705700 2-55834574881' 2026-02-22T20:56:43.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:43.265 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836518 2026-02-22T20:56:43.265 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:43.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:56:43.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836518 2026-02-22T20:56:43.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:43.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836518 2026-02-22T20:56:43.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2026-02-22T20:56:43.268 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836518 2026-02-22T20:56:43.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:43.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836518 2026-02-22T20:56:43.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:56:44.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:56:44.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:44.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836519 -lt 21474836518 2026-02-22T20:56:44.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:44.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705700 2026-02-22T20:56:44.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:44.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:56:44.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705700 2026-02-22T20:56:44.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:44.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705700 2026-02-22T20:56:44.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705700' 2026-02-22T20:56:44.590 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705700 2026-02-22T20:56:44.590 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:56:44.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705700 -lt 38654705700 2026-02-22T20:56:44.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:44.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574881 2026-02-22T20:56:44.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:44.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:56:44.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574881 2026-02-22T20:56:44.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:44.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574881 2026-02-22T20:56:44.757 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574881 2026-02-22T20:56:44.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574881' 2026-02-22T20:56:44.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:56:44.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574881 -lt 55834574881 2026-02-22T20:56:44.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: ceph pg dump pgs 2026-02-22T20:56:44.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep '^1.0' 2026-02-22T20:56:44.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep -q scrubbing 2026-02-22T20:56:45.062 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:56:45.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: found=yes 2026-02-22T20:56:45.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: break 2026-02-22T20:56:45.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: set +o pipefail 2026-02-22T20:56:45.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: test yes = no 2026-02-22T20:56:45.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:362: _scrub_abort: ceph osd set nodeep-scrub 2026-02-22T20:56:45.272 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T20:56:45.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2026-02-22T20:56:45.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:365: _scrub_abort: ceph osd set noscrub 2026-02-22T20:56:45.476 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T20:56:45.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:369: _scrub_abort: set -o pipefail 2026-02-22T20:56:45.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: seq 0 200 2026-02-22T20:56:45.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:56:45.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:56:45.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:56:45.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:56:45.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:56:45.647 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:56:45.647 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:56:45.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:56:45.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:45.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:56:45.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836522 2026-02-22T20:56:45.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836522 2026-02-22T20:56:45.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836522' 2026-02-22T20:56:45.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:45.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:56:45.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705703 2026-02-22T20:56:45.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705703 2026-02-22T20:56:45.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836522 1-38654705703' 2026-02-22T20:56:45.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:45.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:56:45.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574885 2026-02-22T20:56:45.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574885 2026-02-22T20:56:45.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836522 1-38654705703 2-55834574885' 2026-02-22T20:56:45.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:45.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836522 2026-02-22T20:56:45.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:45.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:56:45.865 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836522 2026-02-22T20:56:45.866 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:45.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836522 2026-02-22T20:56:45.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2026-02-22T20:56:45.867 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836522 2026-02-22T20:56:45.867 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:46.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836522 -lt 21474836522 2026-02-22T20:56:46.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:46.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705703 2026-02-22T20:56:46.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:46.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:56:46.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705703 2026-02-22T20:56:46.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:46.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705703 2026-02-22T20:56:46.034 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705703 2026-02-22T20:56:46.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705703' 2026-02-22T20:56:46.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:56:46.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705702 -lt 38654705703 2026-02-22T20:56:46.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:56:47.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:56:47.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:56:47.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705702 -lt 38654705703 2026-02-22T20:56:47.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:56:48.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:56:48.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:56:48.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705705 -lt 38654705703 2026-02-22T20:56:48.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:48.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574885 2026-02-22T20:56:48.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:48.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:56:48.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574885 2026-02-22T20:56:48.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:48.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574885 2026-02-22T20:56:48.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574885' 2026-02-22T20:56:48.528 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574885 2026-02-22T20:56:48.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:56:48.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574886 -lt 55834574885 2026-02-22T20:56:48.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:56:48.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:56:48.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:56:48.831 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:56:48.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:56:48.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:56:48.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:56:48.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:56:48.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:56:48.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:56:48.995 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:56:48.995 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:56:48.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:56:48.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:48.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:56:49.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836526 2026-02-22T20:56:49.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836526 2026-02-22T20:56:49.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836526' 2026-02-22T20:56:49.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:49.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:56:49.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705708 2026-02-22T20:56:49.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705708 2026-02-22T20:56:49.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836526 1-38654705708' 2026-02-22T20:56:49.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:49.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:56:49.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574889 2026-02-22T20:56:49.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574889 2026-02-22T20:56:49.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836526 1-38654705708 2-55834574889' 2026-02-22T20:56:49.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:49.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836526 2026-02-22T20:56:49.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:49.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:56:49.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836526 2026-02-22T20:56:49.217 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:49.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836526 2026-02-22T20:56:49.219 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836526 2026-02-22T20:56:49.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2026-02-22T20:56:49.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:49.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836524 -lt 21474836526 2026-02-22T20:56:49.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:56:50.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:56:50.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:50.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836527 -lt 21474836526 2026-02-22T20:56:50.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:50.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705708 2026-02-22T20:56:50.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:50.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:56:50.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705708 2026-02-22T20:56:50.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:50.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705708 2026-02-22T20:56:50.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705708' 2026-02-22T20:56:50.536 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705708 2026-02-22T20:56:50.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:56:50.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705708 -lt 38654705708 2026-02-22T20:56:50.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:50.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574889 2026-02-22T20:56:50.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:50.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:56:50.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574889 2026-02-22T20:56:50.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:50.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574889 2026-02-22T20:56:50.699 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574889 2026-02-22T20:56:50.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574889' 2026-02-22T20:56:50.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:56:50.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574889 -lt 55834574889 2026-02-22T20:56:50.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:56:50.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:56:50.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:56:51.013 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:56:51.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:56:51.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:56:51.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:56:51.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:56:51.023 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:56:51.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:56:51.178 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:56:51.178 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:56:51.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:56:51.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:51.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:56:51.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836529 2026-02-22T20:56:51.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836529 2026-02-22T20:56:51.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836529' 2026-02-22T20:56:51.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:51.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:56:51.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705711 2026-02-22T20:56:51.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705711 2026-02-22T20:56:51.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836529 1-38654705711' 2026-02-22T20:56:51.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:51.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:56:51.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574892 2026-02-22T20:56:51.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574892 2026-02-22T20:56:51.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836529 1-38654705711 2-55834574892' 2026-02-22T20:56:51.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:51.387 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836529 2026-02-22T20:56:51.388 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:51.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:56:51.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836529 2026-02-22T20:56:51.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:51.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836529 2026-02-22T20:56:51.392 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836529 2026-02-22T20:56:51.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2026-02-22T20:56:51.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:51.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836527 -lt 21474836529 2026-02-22T20:56:51.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:56:52.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:56:52.556 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:52.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836530 -lt 21474836529 2026-02-22T20:56:52.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:52.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705711 2026-02-22T20:56:52.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:52.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:56:52.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705711 2026-02-22T20:56:52.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:52.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705711 2026-02-22T20:56:52.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705711' 2026-02-22T20:56:52.719 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705711 2026-02-22T20:56:52.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:56:52.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705711 -lt 38654705711 2026-02-22T20:56:52.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:52.886 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574892 2026-02-22T20:56:52.886 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:52.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:56:52.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574892 2026-02-22T20:56:52.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:52.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574892 2026-02-22T20:56:52.889 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574892 2026-02-22T20:56:52.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574892' 2026-02-22T20:56:52.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:56:53.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574892 -lt 55834574892 2026-02-22T20:56:53.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:56:53.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:56:53.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:56:53.192 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:56:53.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:56:53.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:56:53.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:56:53.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:56:53.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:56:53.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:56:53.361 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:56:53.361 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:56:53.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:56:53.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:53.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:56:53.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836532 2026-02-22T20:56:53.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836532 2026-02-22T20:56:53.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532' 2026-02-22T20:56:53.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:53.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:56:53.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705714 2026-02-22T20:56:53.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705714 2026-02-22T20:56:53.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532 1-38654705714' 2026-02-22T20:56:53.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:53.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:56:53.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574895 2026-02-22T20:56:53.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574895 2026-02-22T20:56:53.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532 1-38654705714 2-55834574895' 2026-02-22T20:56:53.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:53.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836532 2026-02-22T20:56:53.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:53.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:56:53.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836532 2026-02-22T20:56:53.578 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:53.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836532 2026-02-22T20:56:53.579 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836532 2026-02-22T20:56:53.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2026-02-22T20:56:53.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:53.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836530 -lt 21474836532 2026-02-22T20:56:53.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:56:54.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:56:54.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:54.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836533 -lt 21474836532 2026-02-22T20:56:54.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:54.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705714 2026-02-22T20:56:54.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:54.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:56:54.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705714 2026-02-22T20:56:54.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:54.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705714 2026-02-22T20:56:54.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705714' 2026-02-22T20:56:54.905 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705714 2026-02-22T20:56:54.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:56:55.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705714 -lt 38654705714 2026-02-22T20:56:55.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:55.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574895 2026-02-22T20:56:55.071 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:55.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:56:55.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574895 2026-02-22T20:56:55.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:55.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574895 2026-02-22T20:56:55.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574895' 2026-02-22T20:56:55.074 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574895 2026-02-22T20:56:55.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:56:55.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574895 -lt 55834574895 2026-02-22T20:56:55.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:56:55.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:56:55.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:56:55.390 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:56:55.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:56:55.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:56:55.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:56:55.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:56:55.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:56:55.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:56:55.558 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:56:55.558 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:56:55.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:56:55.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:55.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:56:55.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836535 2026-02-22T20:56:55.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836535 2026-02-22T20:56:55.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836535' 2026-02-22T20:56:55.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:55.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:56:55.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705717 2026-02-22T20:56:55.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705717 2026-02-22T20:56:55.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836535 1-38654705717' 2026-02-22T20:56:55.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:55.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:56:55.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574898 2026-02-22T20:56:55.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574898 2026-02-22T20:56:55.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836535 1-38654705717 2-55834574898' 2026-02-22T20:56:55.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:55.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836535 2026-02-22T20:56:55.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:55.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:56:55.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836535 2026-02-22T20:56:55.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:55.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836535 2026-02-22T20:56:55.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2026-02-22T20:56:55.780 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836535 2026-02-22T20:56:55.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:55.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836536 -lt 21474836535 2026-02-22T20:56:55.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:55.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705717 2026-02-22T20:56:55.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:55.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:56:55.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705717 2026-02-22T20:56:55.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:55.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705717 2026-02-22T20:56:55.948 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705717 2026-02-22T20:56:55.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705717' 2026-02-22T20:56:55.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:56:56.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705717 -lt 38654705717 2026-02-22T20:56:56.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:56.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574898 2026-02-22T20:56:56.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:56.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:56:56.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574898 2026-02-22T20:56:56.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:56.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574898 2026-02-22T20:56:56.116 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574898 2026-02-22T20:56:56.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574898' 2026-02-22T20:56:56.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:56:56.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574898 -lt 55834574898 2026-02-22T20:56:56.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:56:56.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:56:56.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:56:56.431 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:56:56.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:56:56.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:56:56.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:56:56.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:56:56.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:56:56.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:56:56.591 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:56:56.591 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:56:56.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:56:56.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:56.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:56:56.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836538 2026-02-22T20:56:56.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836538 2026-02-22T20:56:56.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836538' 2026-02-22T20:56:56.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:56.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:56:56.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705719 2026-02-22T20:56:56.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705719 2026-02-22T20:56:56.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836538 1-38654705719' 2026-02-22T20:56:56.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:56.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:56:56.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574900 2026-02-22T20:56:56.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574900 2026-02-22T20:56:56.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836538 1-38654705719 2-55834574900' 2026-02-22T20:56:56.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:56.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836538 2026-02-22T20:56:56.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:56.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:56:56.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836538 2026-02-22T20:56:56.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:56.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836538 2026-02-22T20:56:56.811 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836538 2026-02-22T20:56:56.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2026-02-22T20:56:56.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:56.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836536 -lt 21474836538 2026-02-22T20:56:56.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:56:57.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:56:57.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:58.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836539 -lt 21474836538 2026-02-22T20:56:58.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:58.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705719 2026-02-22T20:56:58.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:58.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:56:58.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705719 2026-02-22T20:56:58.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:58.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705719 2026-02-22T20:56:58.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705719' 2026-02-22T20:56:58.140 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705719 2026-02-22T20:56:58.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:56:58.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705720 -lt 38654705719 2026-02-22T20:56:58.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:58.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574900 2026-02-22T20:56:58.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:58.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:56:58.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574900 2026-02-22T20:56:58.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:58.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574900 2026-02-22T20:56:58.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574900' 2026-02-22T20:56:58.313 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574900 2026-02-22T20:56:58.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:56:58.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574901 -lt 55834574900 2026-02-22T20:56:58.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:56:58.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:56:58.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:56:58.632 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:56:58.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:56:58.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:56:58.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:56:58.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:56:58.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:56:58.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:56:58.796 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:56:58.796 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:56:58.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:56:58.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:58.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:56:58.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836541 2026-02-22T20:56:58.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836541 2026-02-22T20:56:58.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836541' 2026-02-22T20:56:58.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:58.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:56:58.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705722 2026-02-22T20:56:58.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705722 2026-02-22T20:56:58.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836541 1-38654705722' 2026-02-22T20:56:58.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:56:58.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:56:59.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574904 2026-02-22T20:56:59.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574904 2026-02-22T20:56:59.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836541 1-38654705722 2-55834574904' 2026-02-22T20:56:59.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:56:59.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836541 2026-02-22T20:56:59.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:56:59.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:56:59.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836541 2026-02-22T20:56:59.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:56:59.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836541 2026-02-22T20:56:59.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2026-02-22T20:56:59.016 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836541 2026-02-22T20:56:59.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:56:59.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836539 -lt 21474836541 2026-02-22T20:56:59.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:00.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:00.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:00.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836542 -lt 21474836541 2026-02-22T20:57:00.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:00.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705722 2026-02-22T20:57:00.347 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:00.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:00.349 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705722 2026-02-22T20:57:00.349 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:00.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705722 2026-02-22T20:57:00.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705722' 2026-02-22T20:57:00.351 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705722 2026-02-22T20:57:00.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:00.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705723 -lt 38654705722 2026-02-22T20:57:00.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:00.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574904 2026-02-22T20:57:00.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:00.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:00.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574904 2026-02-22T20:57:00.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:00.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574904 2026-02-22T20:57:00.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574904' 2026-02-22T20:57:00.518 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574904 2026-02-22T20:57:00.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:00.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574904 -lt 55834574904 2026-02-22T20:57:00.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:00.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:00.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:00.824 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:00.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:00.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:00.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:00.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:00.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:00.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:00.994 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:00.994 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:00.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:00.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:00.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:01.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836544 2026-02-22T20:57:01.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836544 2026-02-22T20:57:01.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836544' 2026-02-22T20:57:01.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:01.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:01.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705726 2026-02-22T20:57:01.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705726 2026-02-22T20:57:01.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836544 1-38654705726' 2026-02-22T20:57:01.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:01.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:01.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574907 2026-02-22T20:57:01.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574907 2026-02-22T20:57:01.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836544 1-38654705726 2-55834574907' 2026-02-22T20:57:01.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:01.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836544 2026-02-22T20:57:01.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:01.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:01.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836544 2026-02-22T20:57:01.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:01.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836544 2026-02-22T20:57:01.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2026-02-22T20:57:01.216 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836544 2026-02-22T20:57:01.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:01.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836542 -lt 21474836544 2026-02-22T20:57:01.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:02.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:02.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:02.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836545 -lt 21474836544 2026-02-22T20:57:02.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:02.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705726 2026-02-22T20:57:02.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:02.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:02.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705726 2026-02-22T20:57:02.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:02.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705726 2026-02-22T20:57:02.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705726' 2026-02-22T20:57:02.543 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705726 2026-02-22T20:57:02.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:02.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705726 -lt 38654705726 2026-02-22T20:57:02.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:02.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574907 2026-02-22T20:57:02.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:02.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:02.711 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574907 2026-02-22T20:57:02.711 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:02.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574907 2026-02-22T20:57:02.712 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574907 2026-02-22T20:57:02.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574907' 2026-02-22T20:57:02.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:02.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574907 -lt 55834574907 2026-02-22T20:57:02.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:02.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:02.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:03.023 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:03.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:03.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:03.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:03.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:03.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:03.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:03.192 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:03.192 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:03.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:03.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:03.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:03.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836547 2026-02-22T20:57:03.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836547 2026-02-22T20:57:03.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836547' 2026-02-22T20:57:03.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:03.265 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:03.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705729 2026-02-22T20:57:03.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705729 2026-02-22T20:57:03.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836547 1-38654705729' 2026-02-22T20:57:03.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:03.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:03.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574910 2026-02-22T20:57:03.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574910 2026-02-22T20:57:03.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836547 1-38654705729 2-55834574910' 2026-02-22T20:57:03.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:03.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836547 2026-02-22T20:57:03.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:03.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:03.409 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836547 2026-02-22T20:57:03.409 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:03.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836547 2026-02-22T20:57:03.411 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836547 2026-02-22T20:57:03.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2026-02-22T20:57:03.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:03.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836545 -lt 21474836547 2026-02-22T20:57:03.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:04.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:04.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:04.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836548 -lt 21474836547 2026-02-22T20:57:04.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:04.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705729 2026-02-22T20:57:04.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:04.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:04.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705729 2026-02-22T20:57:04.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:04.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705729 2026-02-22T20:57:04.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705729' 2026-02-22T20:57:04.741 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705729 2026-02-22T20:57:04.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:04.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705729 -lt 38654705729 2026-02-22T20:57:04.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:04.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574910 2026-02-22T20:57:04.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:04.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:04.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574910 2026-02-22T20:57:04.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:04.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574910 2026-02-22T20:57:04.905 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574910 2026-02-22T20:57:04.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574910' 2026-02-22T20:57:04.905 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:05.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574910 -lt 55834574910 2026-02-22T20:57:05.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:05.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:05.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:05.212 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:05.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:05.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:05.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:05.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:05.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:05.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:05.377 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:05.377 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:05.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:05.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:05.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:05.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836550 2026-02-22T20:57:05.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836550 2026-02-22T20:57:05.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550' 2026-02-22T20:57:05.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:05.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:05.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705732 2026-02-22T20:57:05.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705732 2026-02-22T20:57:05.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550 1-38654705732' 2026-02-22T20:57:05.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:05.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:05.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574913 2026-02-22T20:57:05.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574913 2026-02-22T20:57:05.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550 1-38654705732 2-55834574913' 2026-02-22T20:57:05.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:05.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836550 2026-02-22T20:57:05.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:05.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:05.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836550 2026-02-22T20:57:05.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:05.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836550 2026-02-22T20:57:05.603 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836550 2026-02-22T20:57:05.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2026-02-22T20:57:05.603 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:05.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836548 -lt 21474836550 2026-02-22T20:57:05.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:06.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:06.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:06.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836551 -lt 21474836550 2026-02-22T20:57:06.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:06.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705732 2026-02-22T20:57:06.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:06.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:06.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705732 2026-02-22T20:57:06.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:06.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705732 2026-02-22T20:57:06.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705732' 2026-02-22T20:57:06.932 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705732 2026-02-22T20:57:06.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:07.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705732 -lt 38654705732 2026-02-22T20:57:07.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:07.099 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574913 2026-02-22T20:57:07.099 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:07.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:07.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574913 2026-02-22T20:57:07.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:07.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574913 2026-02-22T20:57:07.103 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574913 2026-02-22T20:57:07.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574913' 2026-02-22T20:57:07.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:07.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574913 -lt 55834574913 2026-02-22T20:57:07.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:07.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:07.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:07.417 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:07.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:07.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:07.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:07.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:07.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:07.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:07.580 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:07.580 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:07.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:07.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:07.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:07.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836554 2026-02-22T20:57:07.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836554 2026-02-22T20:57:07.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836554' 2026-02-22T20:57:07.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:07.652 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:07.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705735 2026-02-22T20:57:07.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705735 2026-02-22T20:57:07.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836554 1-38654705735' 2026-02-22T20:57:07.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:07.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:07.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574916 2026-02-22T20:57:07.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574916 2026-02-22T20:57:07.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836554 1-38654705735 2-55834574916' 2026-02-22T20:57:07.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:07.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836554 2026-02-22T20:57:07.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:07.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:07.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836554 2026-02-22T20:57:07.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:07.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836554 2026-02-22T20:57:07.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2026-02-22T20:57:07.799 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836554 2026-02-22T20:57:07.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:07.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836554 -lt 21474836554 2026-02-22T20:57:07.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:07.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705735 2026-02-22T20:57:07.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:07.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:07.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705735 2026-02-22T20:57:07.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:07.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705735 2026-02-22T20:57:07.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705735' 2026-02-22T20:57:07.973 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705735 2026-02-22T20:57:07.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:08.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705735 -lt 38654705735 2026-02-22T20:57:08.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:08.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574916 2026-02-22T20:57:08.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:08.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:08.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574916 2026-02-22T20:57:08.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:08.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574916 2026-02-22T20:57:08.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574916' 2026-02-22T20:57:08.138 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574916 2026-02-22T20:57:08.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:08.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574915 -lt 55834574916 2026-02-22T20:57:08.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:09.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:09.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:09.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574915 -lt 55834574916 2026-02-22T20:57:09.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:10.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:57:10.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:10.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574918 -lt 55834574916 2026-02-22T20:57:10.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:10.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:10.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:10.771 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:10.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:10.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:10.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:10.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:10.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:10.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:10.977 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:10.977 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:10.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:10.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:10.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:11.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836558 2026-02-22T20:57:11.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836558 2026-02-22T20:57:11.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558' 2026-02-22T20:57:11.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:11.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:11.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705740 2026-02-22T20:57:11.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705740 2026-02-22T20:57:11.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558 1-38654705740' 2026-02-22T20:57:11.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:11.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:11.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574921 2026-02-22T20:57:11.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574921 2026-02-22T20:57:11.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558 1-38654705740 2-55834574921' 2026-02-22T20:57:11.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:11.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836558 2026-02-22T20:57:11.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:11.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:11.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836558 2026-02-22T20:57:11.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:11.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836558 2026-02-22T20:57:11.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2026-02-22T20:57:11.156 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836558 2026-02-22T20:57:11.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:11.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836556 -lt 21474836558 2026-02-22T20:57:11.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:12.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:12.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:12.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836559 -lt 21474836558 2026-02-22T20:57:12.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:12.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705740 2026-02-22T20:57:12.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:12.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:12.482 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705740 2026-02-22T20:57:12.482 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:12.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705740 2026-02-22T20:57:12.483 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705740 2026-02-22T20:57:12.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705740' 2026-02-22T20:57:12.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:12.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705740 -lt 38654705740 2026-02-22T20:57:12.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:12.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574921 2026-02-22T20:57:12.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:12.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:12.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574921 2026-02-22T20:57:12.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:12.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574921 2026-02-22T20:57:12.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574921' 2026-02-22T20:57:12.658 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574921 2026-02-22T20:57:12.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:12.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574921 -lt 55834574921 2026-02-22T20:57:12.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:12.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:12.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:12.973 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:12.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:12.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:12.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:12.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:12.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:13.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:13.141 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:13.141 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:13.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:13.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:13.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:13.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836561 2026-02-22T20:57:13.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836561 2026-02-22T20:57:13.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561' 2026-02-22T20:57:13.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:13.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:13.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705743 2026-02-22T20:57:13.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705743 2026-02-22T20:57:13.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561 1-38654705743' 2026-02-22T20:57:13.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:13.287 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:13.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574924 2026-02-22T20:57:13.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574924 2026-02-22T20:57:13.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561 1-38654705743 2-55834574924' 2026-02-22T20:57:13.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:13.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836561 2026-02-22T20:57:13.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:13.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:13.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836561 2026-02-22T20:57:13.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:13.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836561 2026-02-22T20:57:13.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2026-02-22T20:57:13.366 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836561 2026-02-22T20:57:13.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:13.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836559 -lt 21474836561 2026-02-22T20:57:13.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:14.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:14.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:14.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836562 -lt 21474836561 2026-02-22T20:57:14.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:14.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705743 2026-02-22T20:57:14.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:14.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:14.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705743 2026-02-22T20:57:14.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:14.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705743 2026-02-22T20:57:14.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705743' 2026-02-22T20:57:14.689 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705743 2026-02-22T20:57:14.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:14.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705743 -lt 38654705743 2026-02-22T20:57:14.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:14.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574924 2026-02-22T20:57:14.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:14.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:14.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574924 2026-02-22T20:57:14.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:14.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574924 2026-02-22T20:57:14.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574924' 2026-02-22T20:57:14.858 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574924 2026-02-22T20:57:14.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:15.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574924 -lt 55834574924 2026-02-22T20:57:15.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:15.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:15.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:15.185 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:15.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:15.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:15.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:15.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:15.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:15.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:15.351 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:15.351 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:15.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:15.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:15.352 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:15.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836564 2026-02-22T20:57:15.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836564 2026-02-22T20:57:15.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564' 2026-02-22T20:57:15.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:15.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:15.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705746 2026-02-22T20:57:15.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705746 2026-02-22T20:57:15.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564 1-38654705746' 2026-02-22T20:57:15.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:15.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:15.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574927 2026-02-22T20:57:15.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574927 2026-02-22T20:57:15.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564 1-38654705746 2-55834574927' 2026-02-22T20:57:15.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:15.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836564 2026-02-22T20:57:15.567 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:15.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:15.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836564 2026-02-22T20:57:15.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:15.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836564 2026-02-22T20:57:15.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2026-02-22T20:57:15.571 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836564 2026-02-22T20:57:15.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:15.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836562 -lt 21474836564 2026-02-22T20:57:15.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:16.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:16.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:16.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836565 -lt 21474836564 2026-02-22T20:57:16.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:16.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705746 2026-02-22T20:57:16.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:16.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:16.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705746 2026-02-22T20:57:16.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:16.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705746 2026-02-22T20:57:16.902 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705746 2026-02-22T20:57:16.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705746' 2026-02-22T20:57:16.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:17.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705746 -lt 38654705746 2026-02-22T20:57:17.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:17.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574927 2026-02-22T20:57:17.066 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:17.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:17.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574927 2026-02-22T20:57:17.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:17.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574927 2026-02-22T20:57:17.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574927' 2026-02-22T20:57:17.069 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574927 2026-02-22T20:57:17.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:17.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574927 -lt 55834574927 2026-02-22T20:57:17.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:17.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:17.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:17.376 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:17.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:17.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:17.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:17.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:17.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:17.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:17.539 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:17.539 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:17.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:17.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:17.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:17.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836567 2026-02-22T20:57:17.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836567 2026-02-22T20:57:17.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836567' 2026-02-22T20:57:17.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:17.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:17.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705749 2026-02-22T20:57:17.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705749 2026-02-22T20:57:17.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836567 1-38654705749' 2026-02-22T20:57:17.686 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:17.686 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:17.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574930 2026-02-22T20:57:17.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574930 2026-02-22T20:57:17.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836567 1-38654705749 2-55834574930' 2026-02-22T20:57:17.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:17.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836567 2026-02-22T20:57:17.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:17.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:17.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836567 2026-02-22T20:57:17.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:17.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836567 2026-02-22T20:57:17.764 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836567 2026-02-22T20:57:17.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2026-02-22T20:57:17.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:17.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836568 -lt 21474836567 2026-02-22T20:57:17.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:17.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705749 2026-02-22T20:57:17.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:17.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:17.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705749 2026-02-22T20:57:17.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:17.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705749 2026-02-22T20:57:17.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705749' 2026-02-22T20:57:17.936 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705749 2026-02-22T20:57:17.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:18.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705749 -lt 38654705749 2026-02-22T20:57:18.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:18.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574930 2026-02-22T20:57:18.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:18.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:18.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574930 2026-02-22T20:57:18.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:18.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574930 2026-02-22T20:57:18.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574930' 2026-02-22T20:57:18.099 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574930 2026-02-22T20:57:18.099 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:18.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574930 -lt 55834574930 2026-02-22T20:57:18.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:18.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:18.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:18.412 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:18.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:18.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:18.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:18.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:18.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:18.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:18.580 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:18.580 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:18.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:18.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:18.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:18.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836570 2026-02-22T20:57:18.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836570 2026-02-22T20:57:18.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836570' 2026-02-22T20:57:18.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:18.652 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:18.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705751 2026-02-22T20:57:18.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705751 2026-02-22T20:57:18.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836570 1-38654705751' 2026-02-22T20:57:18.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:18.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:18.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574932 2026-02-22T20:57:18.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574932 2026-02-22T20:57:18.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836570 1-38654705751 2-55834574932' 2026-02-22T20:57:18.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:18.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836570 2026-02-22T20:57:18.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:18.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:18.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836570 2026-02-22T20:57:18.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:18.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836570 2026-02-22T20:57:18.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2026-02-22T20:57:18.796 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836570 2026-02-22T20:57:18.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:18.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836568 -lt 21474836570 2026-02-22T20:57:18.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:19.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:19.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:20.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836571 -lt 21474836570 2026-02-22T20:57:20.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:20.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705751 2026-02-22T20:57:20.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:20.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:20.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705751 2026-02-22T20:57:20.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:20.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705751 2026-02-22T20:57:20.124 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705751 2026-02-22T20:57:20.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705751' 2026-02-22T20:57:20.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:20.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705752 -lt 38654705751 2026-02-22T20:57:20.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:20.282 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574932 2026-02-22T20:57:20.282 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:20.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:20.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574932 2026-02-22T20:57:20.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:20.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574932 2026-02-22T20:57:20.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574932' 2026-02-22T20:57:20.286 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574932 2026-02-22T20:57:20.286 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:20.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574933 -lt 55834574932 2026-02-22T20:57:20.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:20.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:20.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:20.602 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:20.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:20.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:20.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:20.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:20.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:20.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:20.762 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:20.763 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:20.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:20.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:20.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:20.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836573 2026-02-22T20:57:20.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836573 2026-02-22T20:57:20.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836573' 2026-02-22T20:57:20.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:20.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:20.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705754 2026-02-22T20:57:20.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705754 2026-02-22T20:57:20.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836573 1-38654705754' 2026-02-22T20:57:20.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:20.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:20.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574936 2026-02-22T20:57:20.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574936 2026-02-22T20:57:20.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836573 1-38654705754 2-55834574936' 2026-02-22T20:57:20.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:20.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836573 2026-02-22T20:57:20.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:20.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:20.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836573 2026-02-22T20:57:20.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:20.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836573 2026-02-22T20:57:20.984 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836573 2026-02-22T20:57:20.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2026-02-22T20:57:20.985 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:21.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836571 -lt 21474836573 2026-02-22T20:57:21.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:22.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:22.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:22.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836574 -lt 21474836573 2026-02-22T20:57:22.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:22.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705754 2026-02-22T20:57:22.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:22.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:22.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705754 2026-02-22T20:57:22.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:22.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705754 2026-02-22T20:57:22.312 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705754 2026-02-22T20:57:22.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705754' 2026-02-22T20:57:22.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:22.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705755 -lt 38654705754 2026-02-22T20:57:22.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:22.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574936 2026-02-22T20:57:22.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:22.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:22.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574936 2026-02-22T20:57:22.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:22.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574936 2026-02-22T20:57:22.487 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574936 2026-02-22T20:57:22.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574936' 2026-02-22T20:57:22.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:22.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574936 -lt 55834574936 2026-02-22T20:57:22.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:22.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:22.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:22.804 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:22.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:22.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:22.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:22.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:22.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:22.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:22.970 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:22.970 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:22.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:22.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:22.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:23.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836576 2026-02-22T20:57:23.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836576 2026-02-22T20:57:23.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836576' 2026-02-22T20:57:23.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:23.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:23.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705758 2026-02-22T20:57:23.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705758 2026-02-22T20:57:23.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836576 1-38654705758' 2026-02-22T20:57:23.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:23.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:23.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574939 2026-02-22T20:57:23.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574939 2026-02-22T20:57:23.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836576 1-38654705758 2-55834574939' 2026-02-22T20:57:23.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:23.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836576 2026-02-22T20:57:23.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:23.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:23.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836576 2026-02-22T20:57:23.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:23.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836576 2026-02-22T20:57:23.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2026-02-22T20:57:23.195 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836576 2026-02-22T20:57:23.195 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:23.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836574 -lt 21474836576 2026-02-22T20:57:23.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:24.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:24.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:24.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836577 -lt 21474836576 2026-02-22T20:57:24.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:24.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705758 2026-02-22T20:57:24.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:24.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:24.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705758 2026-02-22T20:57:24.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:24.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705758 2026-02-22T20:57:24.535 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705758 2026-02-22T20:57:24.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705758' 2026-02-22T20:57:24.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:24.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705758 -lt 38654705758 2026-02-22T20:57:24.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:24.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574939 2026-02-22T20:57:24.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:24.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:24.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574939 2026-02-22T20:57:24.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:24.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574939 2026-02-22T20:57:24.693 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574939 2026-02-22T20:57:24.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574939' 2026-02-22T20:57:24.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:24.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574939 -lt 55834574939 2026-02-22T20:57:24.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:24.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:24.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:25.001 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:25.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:25.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:25.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:25.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:25.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:25.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:25.168 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:25.168 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:25.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:25.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:25.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:25.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836579 2026-02-22T20:57:25.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836579 2026-02-22T20:57:25.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836579' 2026-02-22T20:57:25.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:25.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:25.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705761 2026-02-22T20:57:25.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705761 2026-02-22T20:57:25.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836579 1-38654705761' 2026-02-22T20:57:25.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:25.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:25.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574942 2026-02-22T20:57:25.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574942 2026-02-22T20:57:25.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836579 1-38654705761 2-55834574942' 2026-02-22T20:57:25.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:25.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836579 2026-02-22T20:57:25.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:25.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:25.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836579 2026-02-22T20:57:25.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:25.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836579 2026-02-22T20:57:25.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2026-02-22T20:57:25.386 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836579 2026-02-22T20:57:25.387 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:25.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836577 -lt 21474836579 2026-02-22T20:57:25.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:26.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:26.552 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:26.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836580 -lt 21474836579 2026-02-22T20:57:26.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:26.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705761 2026-02-22T20:57:26.707 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:26.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:26.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705761 2026-02-22T20:57:26.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:26.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705761 2026-02-22T20:57:26.711 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705761 2026-02-22T20:57:26.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705761' 2026-02-22T20:57:26.711 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:26.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705761 -lt 38654705761 2026-02-22T20:57:26.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:26.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574942 2026-02-22T20:57:26.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:26.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:26.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574942 2026-02-22T20:57:26.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:26.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574942 2026-02-22T20:57:26.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574942' 2026-02-22T20:57:26.881 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574942 2026-02-22T20:57:26.882 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:27.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574942 -lt 55834574942 2026-02-22T20:57:27.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:27.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:27.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:27.190 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:27.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:27.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:27.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:27.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:27.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:27.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:27.361 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:27.362 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:27.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:27.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:27.362 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:27.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836582 2026-02-22T20:57:27.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836582 2026-02-22T20:57:27.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582' 2026-02-22T20:57:27.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:27.435 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:27.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705764 2026-02-22T20:57:27.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705764 2026-02-22T20:57:27.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582 1-38654705764' 2026-02-22T20:57:27.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:27.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:27.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574945 2026-02-22T20:57:27.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574945 2026-02-22T20:57:27.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582 1-38654705764 2-55834574945' 2026-02-22T20:57:27.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:27.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836582 2026-02-22T20:57:27.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:27.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:27.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836582 2026-02-22T20:57:27.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:27.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836582 2026-02-22T20:57:27.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2026-02-22T20:57:27.583 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836582 2026-02-22T20:57:27.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:27.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836580 -lt 21474836582 2026-02-22T20:57:27.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:28.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:28.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:28.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836583 -lt 21474836582 2026-02-22T20:57:28.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:28.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705764 2026-02-22T20:57:28.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:28.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:28.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705764 2026-02-22T20:57:28.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:28.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705764 2026-02-22T20:57:28.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705764' 2026-02-22T20:57:28.912 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705764 2026-02-22T20:57:28.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:29.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705764 -lt 38654705764 2026-02-22T20:57:29.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:29.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574945 2026-02-22T20:57:29.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:29.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:29.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574945 2026-02-22T20:57:29.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:29.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574945 2026-02-22T20:57:29.089 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574945 2026-02-22T20:57:29.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574945' 2026-02-22T20:57:29.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:29.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574945 -lt 55834574945 2026-02-22T20:57:29.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:29.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:29.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:29.404 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:29.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:29.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:29.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:29.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:29.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:29.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:29.572 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:29.572 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:29.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:29.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:29.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:29.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836585 2026-02-22T20:57:29.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836585 2026-02-22T20:57:29.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836585' 2026-02-22T20:57:29.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:29.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:29.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705767 2026-02-22T20:57:29.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705767 2026-02-22T20:57:29.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836585 1-38654705767' 2026-02-22T20:57:29.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:29.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:29.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574948 2026-02-22T20:57:29.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574948 2026-02-22T20:57:29.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836585 1-38654705767 2-55834574948' 2026-02-22T20:57:29.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:29.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836585 2026-02-22T20:57:29.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:29.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:29.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836585 2026-02-22T20:57:29.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:29.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836585 2026-02-22T20:57:29.792 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836585 2026-02-22T20:57:29.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2026-02-22T20:57:29.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:29.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836586 -lt 21474836585 2026-02-22T20:57:29.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:29.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705767 2026-02-22T20:57:29.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:29.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:29.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705767 2026-02-22T20:57:29.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:29.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705767 2026-02-22T20:57:29.968 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705767 2026-02-22T20:57:29.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705767' 2026-02-22T20:57:29.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:30.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705767 -lt 38654705767 2026-02-22T20:57:30.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:30.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574948 2026-02-22T20:57:30.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:30.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:30.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574948 2026-02-22T20:57:30.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:30.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574948 2026-02-22T20:57:30.141 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574948 2026-02-22T20:57:30.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574948' 2026-02-22T20:57:30.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:30.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574948 -lt 55834574948 2026-02-22T20:57:30.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:30.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:30.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:30.450 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:30.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:30.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:30.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:30.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:30.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:30.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:30.615 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:30.615 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:30.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:30.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:30.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:30.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836588 2026-02-22T20:57:30.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836588 2026-02-22T20:57:30.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836588' 2026-02-22T20:57:30.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:30.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:30.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705769 2026-02-22T20:57:30.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705769 2026-02-22T20:57:30.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836588 1-38654705769' 2026-02-22T20:57:30.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:30.762 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:30.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574950 2026-02-22T20:57:30.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574950 2026-02-22T20:57:30.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836588 1-38654705769 2-55834574950' 2026-02-22T20:57:30.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:30.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836588 2026-02-22T20:57:30.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:30.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:30.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836588 2026-02-22T20:57:30.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:30.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836588 2026-02-22T20:57:30.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836588' 2026-02-22T20:57:30.836 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836588 2026-02-22T20:57:30.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:31.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836586 -lt 21474836588 2026-02-22T20:57:31.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:32.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:32.009 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:32.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836589 -lt 21474836588 2026-02-22T20:57:32.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:32.164 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705769 2026-02-22T20:57:32.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:32.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:32.167 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705769 2026-02-22T20:57:32.167 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:32.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705769 2026-02-22T20:57:32.168 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705769 2026-02-22T20:57:32.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705769' 2026-02-22T20:57:32.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:32.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705770 -lt 38654705769 2026-02-22T20:57:32.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:32.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574950 2026-02-22T20:57:32.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:32.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:32.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574950 2026-02-22T20:57:32.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:32.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574950 2026-02-22T20:57:32.335 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574950 2026-02-22T20:57:32.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574950' 2026-02-22T20:57:32.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:32.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574951 -lt 55834574950 2026-02-22T20:57:32.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:32.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:32.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:32.651 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:32.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:32.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:32.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:32.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:32.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:32.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:32.820 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:32.820 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:32.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:32.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:32.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:32.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836591 2026-02-22T20:57:32.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836591 2026-02-22T20:57:32.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836591' 2026-02-22T20:57:32.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:32.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:32.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705772 2026-02-22T20:57:32.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705772 2026-02-22T20:57:32.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836591 1-38654705772' 2026-02-22T20:57:32.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:32.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:33.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574954 2026-02-22T20:57:33.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574954 2026-02-22T20:57:33.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836591 1-38654705772 2-55834574954' 2026-02-22T20:57:33.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:33.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836591 2026-02-22T20:57:33.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:33.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:33.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836591 2026-02-22T20:57:33.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:33.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836591 2026-02-22T20:57:33.039 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836591 2026-02-22T20:57:33.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836591' 2026-02-22T20:57:33.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:33.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836589 -lt 21474836591 2026-02-22T20:57:33.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:34.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:34.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:34.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836592 -lt 21474836591 2026-02-22T20:57:34.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:34.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705772 2026-02-22T20:57:34.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:34.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:34.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705772 2026-02-22T20:57:34.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:34.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705772 2026-02-22T20:57:34.373 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705772 2026-02-22T20:57:34.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705772' 2026-02-22T20:57:34.374 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:34.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705773 -lt 38654705772 2026-02-22T20:57:34.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:34.537 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574954 2026-02-22T20:57:34.537 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:34.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:34.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574954 2026-02-22T20:57:34.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:34.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574954 2026-02-22T20:57:34.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574954' 2026-02-22T20:57:34.540 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574954 2026-02-22T20:57:34.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:34.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574954 -lt 55834574954 2026-02-22T20:57:34.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:34.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:34.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:34.853 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:34.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:34.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:34.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:34.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:34.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:35.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:35.025 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:35.025 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:35.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:35.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:35.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:35.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836594 2026-02-22T20:57:35.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836594 2026-02-22T20:57:35.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836594' 2026-02-22T20:57:35.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:35.099 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:35.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705776 2026-02-22T20:57:35.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705776 2026-02-22T20:57:35.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836594 1-38654705776' 2026-02-22T20:57:35.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:35.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:35.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574957 2026-02-22T20:57:35.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574957 2026-02-22T20:57:35.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836594 1-38654705776 2-55834574957' 2026-02-22T20:57:35.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:35.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836594 2026-02-22T20:57:35.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:35.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:35.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836594 2026-02-22T20:57:35.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:35.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836594 2026-02-22T20:57:35.250 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836594 2026-02-22T20:57:35.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836594' 2026-02-22T20:57:35.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:35.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836592 -lt 21474836594 2026-02-22T20:57:35.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:36.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:36.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:36.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836595 -lt 21474836594 2026-02-22T20:57:36.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:36.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705776 2026-02-22T20:57:36.565 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:36.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:36.566 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705776 2026-02-22T20:57:36.566 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:36.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705776 2026-02-22T20:57:36.568 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705776 2026-02-22T20:57:36.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705776' 2026-02-22T20:57:36.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:36.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705776 -lt 38654705776 2026-02-22T20:57:36.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:36.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574957 2026-02-22T20:57:36.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:36.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:36.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574957 2026-02-22T20:57:36.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:36.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574957 2026-02-22T20:57:36.738 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574957 2026-02-22T20:57:36.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574957' 2026-02-22T20:57:36.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:36.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574957 -lt 55834574957 2026-02-22T20:57:36.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:36.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:36.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:37.053 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:37.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:37.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:37.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:37.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:37.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:37.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:37.218 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:37.218 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:37.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:37.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:37.218 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:37.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836597 2026-02-22T20:57:37.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836597 2026-02-22T20:57:37.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836597' 2026-02-22T20:57:37.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:37.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:37.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705779 2026-02-22T20:57:37.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705779 2026-02-22T20:57:37.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836597 1-38654705779' 2026-02-22T20:57:37.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:37.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:37.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574960 2026-02-22T20:57:37.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574960 2026-02-22T20:57:37.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836597 1-38654705779 2-55834574960' 2026-02-22T20:57:37.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:37.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836597 2026-02-22T20:57:37.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:37.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:37.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836597 2026-02-22T20:57:37.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:37.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836597 2026-02-22T20:57:37.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836597' 2026-02-22T20:57:37.440 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836597 2026-02-22T20:57:37.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:37.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836595 -lt 21474836597 2026-02-22T20:57:37.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:38.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:38.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:38.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836598 -lt 21474836597 2026-02-22T20:57:38.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:38.767 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705779 2026-02-22T20:57:38.767 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:38.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:38.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705779 2026-02-22T20:57:38.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:38.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705779 2026-02-22T20:57:38.771 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705779 2026-02-22T20:57:38.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705779' 2026-02-22T20:57:38.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:38.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705779 -lt 38654705779 2026-02-22T20:57:38.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:38.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574960 2026-02-22T20:57:38.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:38.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:38.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574960 2026-02-22T20:57:38.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:38.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574960 2026-02-22T20:57:38.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574960' 2026-02-22T20:57:38.939 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574960 2026-02-22T20:57:38.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:39.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574960 -lt 55834574960 2026-02-22T20:57:39.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:39.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:39.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:39.253 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:39.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:39.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:39.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:39.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:39.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:39.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:39.425 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:39.425 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:39.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:39.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:39.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:39.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836600 2026-02-22T20:57:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836600 2026-02-22T20:57:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836600' 2026-02-22T20:57:39.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:39.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:39.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705782 2026-02-22T20:57:39.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705782 2026-02-22T20:57:39.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836600 1-38654705782' 2026-02-22T20:57:39.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:39.568 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:39.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574963 2026-02-22T20:57:39.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574963 2026-02-22T20:57:39.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836600 1-38654705782 2-55834574963' 2026-02-22T20:57:39.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:39.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836600 2026-02-22T20:57:39.640 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:39.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:39.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836600 2026-02-22T20:57:39.642 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:39.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836600 2026-02-22T20:57:39.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836600' 2026-02-22T20:57:39.643 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836600 2026-02-22T20:57:39.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:39.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836598 -lt 21474836600 2026-02-22T20:57:39.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:40.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:40.814 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:40.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836601 -lt 21474836600 2026-02-22T20:57:40.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:40.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705782 2026-02-22T20:57:40.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:40.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:40.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705782 2026-02-22T20:57:40.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:40.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705782 2026-02-22T20:57:40.981 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705782 2026-02-22T20:57:40.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705782' 2026-02-22T20:57:40.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:41.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705782 -lt 38654705782 2026-02-22T20:57:41.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:41.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574963 2026-02-22T20:57:41.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:41.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:41.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574963 2026-02-22T20:57:41.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:41.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574963 2026-02-22T20:57:41.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574963' 2026-02-22T20:57:41.146 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574963 2026-02-22T20:57:41.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:41.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574963 -lt 55834574963 2026-02-22T20:57:41.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:41.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:41.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:41.456 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:41.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:41.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:41.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:41.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:41.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:41.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:41.626 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:41.626 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:41.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:41.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:41.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:41.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836604 2026-02-22T20:57:41.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836604 2026-02-22T20:57:41.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836604' 2026-02-22T20:57:41.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:41.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:41.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705785 2026-02-22T20:57:41.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705785 2026-02-22T20:57:41.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836604 1-38654705785' 2026-02-22T20:57:41.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:41.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:41.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574967 2026-02-22T20:57:41.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574967 2026-02-22T20:57:41.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836604 1-38654705785 2-55834574967' 2026-02-22T20:57:41.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:41.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836604 2026-02-22T20:57:41.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:41.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:41.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836604 2026-02-22T20:57:41.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:41.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836604 2026-02-22T20:57:41.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836604' 2026-02-22T20:57:41.850 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836604 2026-02-22T20:57:41.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:42.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836604 -lt 21474836604 2026-02-22T20:57:42.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:42.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705785 2026-02-22T20:57:42.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:42.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:42.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705785 2026-02-22T20:57:42.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:42.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705785 2026-02-22T20:57:42.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705785' 2026-02-22T20:57:42.015 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705785 2026-02-22T20:57:42.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:42.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705785 -lt 38654705785 2026-02-22T20:57:42.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:42.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574967 2026-02-22T20:57:42.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:42.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:42.175 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574967 2026-02-22T20:57:42.175 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:42.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574967 2026-02-22T20:57:42.177 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574967 2026-02-22T20:57:42.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574967' 2026-02-22T20:57:42.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:42.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574965 -lt 55834574967 2026-02-22T20:57:42.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:43.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:43.341 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:43.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574965 -lt 55834574967 2026-02-22T20:57:43.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:44.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:57:44.503 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:44.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574968 -lt 55834574967 2026-02-22T20:57:44.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:44.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:44.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:44.809 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:44.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:44.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:44.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:44.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:44.819 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:44.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:44.970 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:44.970 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:44.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:44.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:44.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:45.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836608 2026-02-22T20:57:45.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836608 2026-02-22T20:57:45.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836608' 2026-02-22T20:57:45.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:45.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:45.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705790 2026-02-22T20:57:45.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705790 2026-02-22T20:57:45.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836608 1-38654705790' 2026-02-22T20:57:45.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:45.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:45.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574971 2026-02-22T20:57:45.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574971 2026-02-22T20:57:45.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836608 1-38654705790 2-55834574971' 2026-02-22T20:57:45.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:45.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836608 2026-02-22T20:57:45.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:45.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:45.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836608 2026-02-22T20:57:45.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:45.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836608 2026-02-22T20:57:45.191 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836608 2026-02-22T20:57:45.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836608' 2026-02-22T20:57:45.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:45.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836606 -lt 21474836608 2026-02-22T20:57:45.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:46.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:46.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:46.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836609 -lt 21474836608 2026-02-22T20:57:46.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:46.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705790 2026-02-22T20:57:46.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:46.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:46.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705790 2026-02-22T20:57:46.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:46.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705790 2026-02-22T20:57:46.519 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705790 2026-02-22T20:57:46.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705790' 2026-02-22T20:57:46.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:46.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705790 -lt 38654705790 2026-02-22T20:57:46.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:46.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574971 2026-02-22T20:57:46.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:46.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:46.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574971 2026-02-22T20:57:46.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:46.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574971 2026-02-22T20:57:46.694 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574971 2026-02-22T20:57:46.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574971' 2026-02-22T20:57:46.694 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:46.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574971 -lt 55834574971 2026-02-22T20:57:46.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:46.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:46.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:47.004 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:47.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:47.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:47.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:47.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:47.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:47.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:47.174 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:47.174 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:47.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:47.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:47.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:47.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836611 2026-02-22T20:57:47.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836611 2026-02-22T20:57:47.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836611' 2026-02-22T20:57:47.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:47.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:47.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705793 2026-02-22T20:57:47.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705793 2026-02-22T20:57:47.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836611 1-38654705793' 2026-02-22T20:57:47.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:47.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:47.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574974 2026-02-22T20:57:47.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574974 2026-02-22T20:57:47.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836611 1-38654705793 2-55834574974' 2026-02-22T20:57:47.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:47.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836611 2026-02-22T20:57:47.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:47.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:47.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836611 2026-02-22T20:57:47.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:47.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836611 2026-02-22T20:57:47.398 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836611 2026-02-22T20:57:47.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836611' 2026-02-22T20:57:47.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:47.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836609 -lt 21474836611 2026-02-22T20:57:47.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:48.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:48.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:48.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836612 -lt 21474836611 2026-02-22T20:57:48.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:48.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705793 2026-02-22T20:57:48.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:48.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:48.721 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705793 2026-02-22T20:57:48.721 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:48.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705793 2026-02-22T20:57:48.722 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705793 2026-02-22T20:57:48.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705793' 2026-02-22T20:57:48.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:48.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705793 -lt 38654705793 2026-02-22T20:57:48.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:48.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574974 2026-02-22T20:57:48.886 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:48.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:48.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574974 2026-02-22T20:57:48.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:48.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574974 2026-02-22T20:57:48.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574974' 2026-02-22T20:57:48.890 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574974 2026-02-22T20:57:48.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:49.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574974 -lt 55834574974 2026-02-22T20:57:49.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:49.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:49.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:49.192 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:49.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:49.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:49.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:49.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:49.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:49.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:49.361 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:49.361 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:49.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:49.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:49.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:49.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836614 2026-02-22T20:57:49.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836614 2026-02-22T20:57:49.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836614' 2026-02-22T20:57:49.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:49.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:49.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705796 2026-02-22T20:57:49.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705796 2026-02-22T20:57:49.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836614 1-38654705796' 2026-02-22T20:57:49.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:49.507 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:49.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574977 2026-02-22T20:57:49.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574977 2026-02-22T20:57:49.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836614 1-38654705796 2-55834574977' 2026-02-22T20:57:49.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:49.582 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836614 2026-02-22T20:57:49.582 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:49.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:49.584 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836614 2026-02-22T20:57:49.584 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:49.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836614 2026-02-22T20:57:49.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836614' 2026-02-22T20:57:49.586 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836614 2026-02-22T20:57:49.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:49.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836612 -lt 21474836614 2026-02-22T20:57:49.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:50.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:50.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:50.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836615 -lt 21474836614 2026-02-22T20:57:50.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:50.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705796 2026-02-22T20:57:50.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:50.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:50.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705796 2026-02-22T20:57:50.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:50.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705796 2026-02-22T20:57:50.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705796' 2026-02-22T20:57:50.908 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705796 2026-02-22T20:57:50.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:51.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705796 -lt 38654705796 2026-02-22T20:57:51.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:51.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574977 2026-02-22T20:57:51.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:51.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:51.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574977 2026-02-22T20:57:51.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:51.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574977 2026-02-22T20:57:51.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574977' 2026-02-22T20:57:51.079 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574977 2026-02-22T20:57:51.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:51.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574977 -lt 55834574977 2026-02-22T20:57:51.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:51.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:51.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:51.386 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:51.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:51.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:51.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:51.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:51.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:51.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:51.553 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:51.553 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:51.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:51.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:51.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:51.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836617 2026-02-22T20:57:51.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836617 2026-02-22T20:57:51.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617' 2026-02-22T20:57:51.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:51.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:51.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705799 2026-02-22T20:57:51.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705799 2026-02-22T20:57:51.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617 1-38654705799' 2026-02-22T20:57:51.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:51.701 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:51.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574980 2026-02-22T20:57:51.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574980 2026-02-22T20:57:51.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617 1-38654705799 2-55834574980' 2026-02-22T20:57:51.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:51.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836617 2026-02-22T20:57:51.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:51.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:51.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836617 2026-02-22T20:57:51.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:51.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836617 2026-02-22T20:57:51.775 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836617 2026-02-22T20:57:51.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836617' 2026-02-22T20:57:51.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:51.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836618 -lt 21474836617 2026-02-22T20:57:51.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:51.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705799 2026-02-22T20:57:51.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:51.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:51.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705799 2026-02-22T20:57:51.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:51.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705799 2026-02-22T20:57:51.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705799' 2026-02-22T20:57:51.939 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705799 2026-02-22T20:57:51.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:52.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705799 -lt 38654705799 2026-02-22T20:57:52.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:52.099 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574980 2026-02-22T20:57:52.099 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:52.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:52.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574980 2026-02-22T20:57:52.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:52.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574980 2026-02-22T20:57:52.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574980' 2026-02-22T20:57:52.102 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574980 2026-02-22T20:57:52.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:52.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574980 -lt 55834574980 2026-02-22T20:57:52.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:52.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:52.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:52.420 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:52.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:52.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:52.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:52.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:52.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:52.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:52.585 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:52.585 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:52.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:52.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:52.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:52.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836620 2026-02-22T20:57:52.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836620 2026-02-22T20:57:52.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836620' 2026-02-22T20:57:52.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:52.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:52.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705801 2026-02-22T20:57:52.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705801 2026-02-22T20:57:52.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836620 1-38654705801' 2026-02-22T20:57:52.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:52.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:52.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574982 2026-02-22T20:57:52.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574982 2026-02-22T20:57:52.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836620 1-38654705801 2-55834574982' 2026-02-22T20:57:52.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:52.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836620 2026-02-22T20:57:52.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:52.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:52.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836620 2026-02-22T20:57:52.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:52.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836620 2026-02-22T20:57:52.804 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836620 2026-02-22T20:57:52.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836620' 2026-02-22T20:57:52.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:52.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836618 -lt 21474836620 2026-02-22T20:57:52.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:53.968 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:53.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:54.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836621 -lt 21474836620 2026-02-22T20:57:54.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:54.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705801 2026-02-22T20:57:54.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:54.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:54.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705801 2026-02-22T20:57:54.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:54.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705801 2026-02-22T20:57:54.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705801' 2026-02-22T20:57:54.135 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705801 2026-02-22T20:57:54.135 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:54.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705802 -lt 38654705801 2026-02-22T20:57:54.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:54.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574982 2026-02-22T20:57:54.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:54.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:54.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574982 2026-02-22T20:57:54.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:54.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574982 2026-02-22T20:57:54.297 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574982 2026-02-22T20:57:54.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574982' 2026-02-22T20:57:54.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:54.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574983 -lt 55834574982 2026-02-22T20:57:54.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:54.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:54.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:54.611 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:54.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:54.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:54.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:54.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:54.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:54.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:54.774 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:54.774 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:54.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:54.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:54.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:54.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836623 2026-02-22T20:57:54.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836623 2026-02-22T20:57:54.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836623' 2026-02-22T20:57:54.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:54.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:54.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705804 2026-02-22T20:57:54.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705804 2026-02-22T20:57:54.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836623 1-38654705804' 2026-02-22T20:57:54.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:54.924 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:54.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574986 2026-02-22T20:57:54.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574986 2026-02-22T20:57:54.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836623 1-38654705804 2-55834574986' 2026-02-22T20:57:54.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:54.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836623 2026-02-22T20:57:54.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:54.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:54.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836623 2026-02-22T20:57:54.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:54.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836623 2026-02-22T20:57:54.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836623' 2026-02-22T20:57:54.999 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836623 2026-02-22T20:57:55.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:55.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836621 -lt 21474836623 2026-02-22T20:57:55.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:56.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:56.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:56.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836624 -lt 21474836623 2026-02-22T20:57:56.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:56.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705804 2026-02-22T20:57:56.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:56.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:56.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705804 2026-02-22T20:57:56.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:56.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705804 2026-02-22T20:57:56.335 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705804 2026-02-22T20:57:56.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705804' 2026-02-22T20:57:56.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:56.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705805 -lt 38654705804 2026-02-22T20:57:56.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:56.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574986 2026-02-22T20:57:56.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:56.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:56.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574986 2026-02-22T20:57:56.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:56.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574986 2026-02-22T20:57:56.508 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574986 2026-02-22T20:57:56.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574986' 2026-02-22T20:57:56.508 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:56.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574986 -lt 55834574986 2026-02-22T20:57:56.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:56.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:56.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:56.826 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:56.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:56.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:56.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:56.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:56.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:56.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:56.996 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:56.996 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:56.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:56.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:56.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:57.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836626 2026-02-22T20:57:57.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836626 2026-02-22T20:57:57.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836626' 2026-02-22T20:57:57.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:57.067 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:57.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705808 2026-02-22T20:57:57.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705808 2026-02-22T20:57:57.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836626 1-38654705808' 2026-02-22T20:57:57.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:57.141 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:57.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574989 2026-02-22T20:57:57.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574989 2026-02-22T20:57:57.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836626 1-38654705808 2-55834574989' 2026-02-22T20:57:57.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:57.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836626 2026-02-22T20:57:57.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:57.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:57.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836626 2026-02-22T20:57:57.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:57.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836626 2026-02-22T20:57:57.215 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836626 2026-02-22T20:57:57.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836626' 2026-02-22T20:57:57.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:57.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836624 -lt 21474836626 2026-02-22T20:57:57.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:57:58.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:57:58.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:58.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836627 -lt 21474836626 2026-02-22T20:57:58.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:58.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705808 2026-02-22T20:57:58.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:58.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:57:58.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705808 2026-02-22T20:57:58.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:58.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705808 2026-02-22T20:57:58.544 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705808 2026-02-22T20:57:58.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705808' 2026-02-22T20:57:58.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:57:58.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705808 -lt 38654705808 2026-02-22T20:57:58.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:58.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574989 2026-02-22T20:57:58.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:58.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:57:58.712 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574989 2026-02-22T20:57:58.712 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:58.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574989 2026-02-22T20:57:58.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574989' 2026-02-22T20:57:58.713 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574989 2026-02-22T20:57:58.714 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:57:58.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574989 -lt 55834574989 2026-02-22T20:57:58.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:57:58.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:57:58.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:57:59.023 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:57:59.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:57:59.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:57:59.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:57:59.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:57:59.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:57:59.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:57:59.187 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:57:59.188 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:57:59.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:57:59.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:59.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:57:59.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836629 2026-02-22T20:57:59.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836629 2026-02-22T20:57:59.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836629' 2026-02-22T20:57:59.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:59.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:57:59.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705811 2026-02-22T20:57:59.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705811 2026-02-22T20:57:59.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836629 1-38654705811' 2026-02-22T20:57:59.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:57:59.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:57:59.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574992 2026-02-22T20:57:59.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574992 2026-02-22T20:57:59.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836629 1-38654705811 2-55834574992' 2026-02-22T20:57:59.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:57:59.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836629 2026-02-22T20:57:59.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:57:59.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:57:59.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836629 2026-02-22T20:57:59.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:57:59.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836629 2026-02-22T20:57:59.408 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836629 2026-02-22T20:57:59.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836629' 2026-02-22T20:57:59.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:57:59.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836627 -lt 21474836629 2026-02-22T20:57:59.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:00.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:00.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:00.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836630 -lt 21474836629 2026-02-22T20:58:00.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:00.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705811 2026-02-22T20:58:00.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:00.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:00.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705811 2026-02-22T20:58:00.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:00.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705811 2026-02-22T20:58:00.738 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705811 2026-02-22T20:58:00.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705811' 2026-02-22T20:58:00.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:00.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705811 -lt 38654705811 2026-02-22T20:58:00.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:00.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574992 2026-02-22T20:58:00.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:00.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:00.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574992 2026-02-22T20:58:00.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:00.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574992 2026-02-22T20:58:00.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574992' 2026-02-22T20:58:00.912 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574992 2026-02-22T20:58:00.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:01.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574992 -lt 55834574992 2026-02-22T20:58:01.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:01.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:01.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:01.225 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:01.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:01.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:01.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:01.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:01.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:01.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:01.395 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:01.396 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:01.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:01.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:01.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:01.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836632 2026-02-22T20:58:01.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836632 2026-02-22T20:58:01.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836632' 2026-02-22T20:58:01.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:01.467 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:01.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705814 2026-02-22T20:58:01.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705814 2026-02-22T20:58:01.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836632 1-38654705814' 2026-02-22T20:58:01.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:01.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:01.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574995 2026-02-22T20:58:01.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574995 2026-02-22T20:58:01.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836632 1-38654705814 2-55834574995' 2026-02-22T20:58:01.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:01.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836632 2026-02-22T20:58:01.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:01.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:01.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836632 2026-02-22T20:58:01.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:01.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836632 2026-02-22T20:58:01.619 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836632 2026-02-22T20:58:01.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836632' 2026-02-22T20:58:01.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:01.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836630 -lt 21474836632 2026-02-22T20:58:01.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:02.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:02.787 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:02.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836633 -lt 21474836632 2026-02-22T20:58:02.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:02.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705814 2026-02-22T20:58:02.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:02.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:02.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705814 2026-02-22T20:58:02.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:02.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705814 2026-02-22T20:58:02.956 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705814 2026-02-22T20:58:02.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705814' 2026-02-22T20:58:02.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:03.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705814 -lt 38654705814 2026-02-22T20:58:03.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:03.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574995 2026-02-22T20:58:03.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:03.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:03.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574995 2026-02-22T20:58:03.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:03.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574995 2026-02-22T20:58:03.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574995' 2026-02-22T20:58:03.125 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574995 2026-02-22T20:58:03.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:03.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574995 -lt 55834574995 2026-02-22T20:58:03.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:03.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:03.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:03.434 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:03.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:03.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:03.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:03.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:03.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:03.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:03.603 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:03.603 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:03.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:03.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:03.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:03.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836636 2026-02-22T20:58:03.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836636 2026-02-22T20:58:03.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836636' 2026-02-22T20:58:03.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:03.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:03.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705817 2026-02-22T20:58:03.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705817 2026-02-22T20:58:03.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836636 1-38654705817' 2026-02-22T20:58:03.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:03.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:03.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574998 2026-02-22T20:58:03.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574998 2026-02-22T20:58:03.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836636 1-38654705817 2-55834574998' 2026-02-22T20:58:03.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:03.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836636 2026-02-22T20:58:03.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:03.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:03.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836636 2026-02-22T20:58:03.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:03.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836636 2026-02-22T20:58:03.824 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836636 2026-02-22T20:58:03.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836636' 2026-02-22T20:58:03.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:03.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836636 -lt 21474836636 2026-02-22T20:58:03.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:03.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705817 2026-02-22T20:58:03.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:03.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:03.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705817 2026-02-22T20:58:03.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:03.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705817 2026-02-22T20:58:03.993 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705817 2026-02-22T20:58:03.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705817' 2026-02-22T20:58:03.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:04.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705817 -lt 38654705817 2026-02-22T20:58:04.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:04.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574998 2026-02-22T20:58:04.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:04.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:04.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834574998 2026-02-22T20:58:04.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:04.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834574998 2026-02-22T20:58:04.160 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834574998 2026-02-22T20:58:04.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834574998' 2026-02-22T20:58:04.160 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:04.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574997 -lt 55834574998 2026-02-22T20:58:04.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:05.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:05.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:05.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834574997 -lt 55834574998 2026-02-22T20:58:05.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:06.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:58:06.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:06.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575000 -lt 55834574998 2026-02-22T20:58:06.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:06.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:06.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:06.805 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:06.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:06.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:06.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:06.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:06.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:06.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:06.979 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:06.979 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:06.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:06.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:06.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:07.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836640 2026-02-22T20:58:07.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836640 2026-02-22T20:58:07.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640' 2026-02-22T20:58:07.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:07.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:07.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705822 2026-02-22T20:58:07.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705822 2026-02-22T20:58:07.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640 1-38654705822' 2026-02-22T20:58:07.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:07.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:07.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575003 2026-02-22T20:58:07.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575003 2026-02-22T20:58:07.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640 1-38654705822 2-55834575003' 2026-02-22T20:58:07.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:07.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836640 2026-02-22T20:58:07.199 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:07.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:07.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836640 2026-02-22T20:58:07.201 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:07.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836640 2026-02-22T20:58:07.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836640' 2026-02-22T20:58:07.203 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836640 2026-02-22T20:58:07.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:07.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836638 -lt 21474836640 2026-02-22T20:58:07.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:08.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:08.374 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:08.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836641 -lt 21474836640 2026-02-22T20:58:08.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:08.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705822 2026-02-22T20:58:08.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:08.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:08.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705822 2026-02-22T20:58:08.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:08.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705822 2026-02-22T20:58:08.534 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705822 2026-02-22T20:58:08.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705822' 2026-02-22T20:58:08.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:08.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705822 -lt 38654705822 2026-02-22T20:58:08.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:08.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575003 2026-02-22T20:58:08.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:08.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:08.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575003 2026-02-22T20:58:08.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:08.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575003 2026-02-22T20:58:08.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575003' 2026-02-22T20:58:08.701 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575003 2026-02-22T20:58:08.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:08.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575003 -lt 55834575003 2026-02-22T20:58:08.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:08.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:08.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:09.011 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:09.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:09.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:09.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:09.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:09.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:09.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:09.178 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:09.178 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:09.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:09.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:09.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:09.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836643 2026-02-22T20:58:09.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836643 2026-02-22T20:58:09.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643' 2026-02-22T20:58:09.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:09.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:09.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705825 2026-02-22T20:58:09.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705825 2026-02-22T20:58:09.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643 1-38654705825' 2026-02-22T20:58:09.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:09.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:09.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575006 2026-02-22T20:58:09.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575006 2026-02-22T20:58:09.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643 1-38654705825 2-55834575006' 2026-02-22T20:58:09.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:09.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836643 2026-02-22T20:58:09.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:09.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:09.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836643 2026-02-22T20:58:09.395 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:09.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836643 2026-02-22T20:58:09.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836643' 2026-02-22T20:58:09.396 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836643 2026-02-22T20:58:09.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:09.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836641 -lt 21474836643 2026-02-22T20:58:09.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:10.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:10.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:10.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836644 -lt 21474836643 2026-02-22T20:58:10.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:10.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705825 2026-02-22T20:58:10.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:10.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:10.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705825 2026-02-22T20:58:10.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:10.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705825 2026-02-22T20:58:10.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705825' 2026-02-22T20:58:10.728 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705825 2026-02-22T20:58:10.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:10.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705825 -lt 38654705825 2026-02-22T20:58:10.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:10.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575006 2026-02-22T20:58:10.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:10.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:10.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575006 2026-02-22T20:58:10.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:10.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575006 2026-02-22T20:58:10.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575006' 2026-02-22T20:58:10.901 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575006 2026-02-22T20:58:10.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:11.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575006 -lt 55834575006 2026-02-22T20:58:11.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:11.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:11.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:11.219 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:11.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:11.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:11.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:11.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:11.229 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:11.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:11.387 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:11.387 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:11.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:11.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:11.387 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:11.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836646 2026-02-22T20:58:11.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836646 2026-02-22T20:58:11.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646' 2026-02-22T20:58:11.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:11.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:11.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705828 2026-02-22T20:58:11.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705828 2026-02-22T20:58:11.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646 1-38654705828' 2026-02-22T20:58:11.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:11.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:11.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575009 2026-02-22T20:58:11.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575009 2026-02-22T20:58:11.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646 1-38654705828 2-55834575009' 2026-02-22T20:58:11.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:11.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836646 2026-02-22T20:58:11.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:11.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:11.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836646 2026-02-22T20:58:11.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:11.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836646 2026-02-22T20:58:11.603 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836646 2026-02-22T20:58:11.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836646' 2026-02-22T20:58:11.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:11.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836644 -lt 21474836646 2026-02-22T20:58:11.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:12.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:12.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:12.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836647 -lt 21474836646 2026-02-22T20:58:12.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:12.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705828 2026-02-22T20:58:12.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:12.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:12.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705828 2026-02-22T20:58:12.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:12.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705828 2026-02-22T20:58:12.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705828' 2026-02-22T20:58:12.933 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705828 2026-02-22T20:58:12.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:13.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705828 -lt 38654705828 2026-02-22T20:58:13.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:13.093 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575009 2026-02-22T20:58:13.093 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:13.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:13.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575009 2026-02-22T20:58:13.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:13.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575009 2026-02-22T20:58:13.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575009' 2026-02-22T20:58:13.097 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575009 2026-02-22T20:58:13.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:13.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575009 -lt 55834575009 2026-02-22T20:58:13.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:13.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:13.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:13.406 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:13.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:13.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:13.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:13.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:13.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:13.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:13.574 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:13.575 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:13.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:13.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:13.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:13.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836649 2026-02-22T20:58:13.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836649 2026-02-22T20:58:13.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836649' 2026-02-22T20:58:13.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:13.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:13.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705831 2026-02-22T20:58:13.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705831 2026-02-22T20:58:13.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836649 1-38654705831' 2026-02-22T20:58:13.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:13.721 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:13.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575012 2026-02-22T20:58:13.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575012 2026-02-22T20:58:13.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836649 1-38654705831 2-55834575012' 2026-02-22T20:58:13.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:13.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836649 2026-02-22T20:58:13.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:13.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:13.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836649 2026-02-22T20:58:13.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:13.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836649 2026-02-22T20:58:13.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836649' 2026-02-22T20:58:13.796 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836649 2026-02-22T20:58:13.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:13.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836650 -lt 21474836649 2026-02-22T20:58:13.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:13.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705831 2026-02-22T20:58:13.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:13.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:13.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705831 2026-02-22T20:58:13.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:13.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705831 2026-02-22T20:58:13.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705831' 2026-02-22T20:58:13.959 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705831 2026-02-22T20:58:13.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:14.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705831 -lt 38654705831 2026-02-22T20:58:14.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:14.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575012 2026-02-22T20:58:14.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:14.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:14.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575012 2026-02-22T20:58:14.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:14.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575012 2026-02-22T20:58:14.130 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575012 2026-02-22T20:58:14.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575012' 2026-02-22T20:58:14.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:14.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575012 -lt 55834575012 2026-02-22T20:58:14.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:14.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:14.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:14.444 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:14.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:14.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:14.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:14.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:14.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:14.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:14.616 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:14.616 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:14.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:14.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:14.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:14.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836652 2026-02-22T20:58:14.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836652 2026-02-22T20:58:14.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836652' 2026-02-22T20:58:14.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:14.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:14.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705833 2026-02-22T20:58:14.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705833 2026-02-22T20:58:14.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836652 1-38654705833' 2026-02-22T20:58:14.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:14.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:14.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575014 2026-02-22T20:58:14.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575014 2026-02-22T20:58:14.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836652 1-38654705833 2-55834575014' 2026-02-22T20:58:14.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:14.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836652 2026-02-22T20:58:14.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:14.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:14.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836652 2026-02-22T20:58:14.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:14.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836652 2026-02-22T20:58:14.835 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836652 2026-02-22T20:58:14.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836652' 2026-02-22T20:58:14.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:15.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836650 -lt 21474836652 2026-02-22T20:58:15.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:16.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:16.009 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:16.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836653 -lt 21474836652 2026-02-22T20:58:16.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:16.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705833 2026-02-22T20:58:16.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:16.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:16.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705833 2026-02-22T20:58:16.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:16.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705833 2026-02-22T20:58:16.173 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705833 2026-02-22T20:58:16.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705833' 2026-02-22T20:58:16.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:16.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705834 -lt 38654705833 2026-02-22T20:58:16.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:16.333 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575014 2026-02-22T20:58:16.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:16.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:16.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575014 2026-02-22T20:58:16.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:16.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575014 2026-02-22T20:58:16.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575014' 2026-02-22T20:58:16.337 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575014 2026-02-22T20:58:16.337 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:16.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575015 -lt 55834575014 2026-02-22T20:58:16.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:16.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:16.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:16.650 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:16.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:16.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:16.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:16.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:16.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:16.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:16.816 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:16.816 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:16.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:16.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:16.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:16.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836655 2026-02-22T20:58:16.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836655 2026-02-22T20:58:16.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836655' 2026-02-22T20:58:16.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:16.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:16.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705836 2026-02-22T20:58:16.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705836 2026-02-22T20:58:16.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836655 1-38654705836' 2026-02-22T20:58:16.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:16.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:17.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575018 2026-02-22T20:58:17.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575018 2026-02-22T20:58:17.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836655 1-38654705836 2-55834575018' 2026-02-22T20:58:17.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:17.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836655 2026-02-22T20:58:17.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:17.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:17.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836655 2026-02-22T20:58:17.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:17.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836655 2026-02-22T20:58:17.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836655' 2026-02-22T20:58:17.034 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836655 2026-02-22T20:58:17.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:17.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836653 -lt 21474836655 2026-02-22T20:58:17.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:18.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:18.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:18.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836656 -lt 21474836655 2026-02-22T20:58:18.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:18.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705836 2026-02-22T20:58:18.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:18.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:18.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705836 2026-02-22T20:58:18.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:18.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705836 2026-02-22T20:58:18.368 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705836 2026-02-22T20:58:18.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705836' 2026-02-22T20:58:18.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:18.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705837 -lt 38654705836 2026-02-22T20:58:18.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:18.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575018 2026-02-22T20:58:18.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:18.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:18.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575018 2026-02-22T20:58:18.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:18.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575018 2026-02-22T20:58:18.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575018' 2026-02-22T20:58:18.538 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575018 2026-02-22T20:58:18.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:18.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575018 -lt 55834575018 2026-02-22T20:58:18.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:18.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:18.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:18.846 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:18.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:18.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:18.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:18.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:18.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:19.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:19.015 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:19.015 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:19.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:19.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:19.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:19.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836658 2026-02-22T20:58:19.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836658 2026-02-22T20:58:19.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658' 2026-02-22T20:58:19.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:19.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:19.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705840 2026-02-22T20:58:19.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705840 2026-02-22T20:58:19.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658 1-38654705840' 2026-02-22T20:58:19.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:19.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:19.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575021 2026-02-22T20:58:19.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575021 2026-02-22T20:58:19.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658 1-38654705840 2-55834575021' 2026-02-22T20:58:19.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:19.230 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836658 2026-02-22T20:58:19.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:19.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:19.233 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836658 2026-02-22T20:58:19.233 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:19.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836658 2026-02-22T20:58:19.234 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836658 2026-02-22T20:58:19.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836658' 2026-02-22T20:58:19.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:19.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836656 -lt 21474836658 2026-02-22T20:58:19.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:20.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:20.395 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:20.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836659 -lt 21474836658 2026-02-22T20:58:20.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:20.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705840 2026-02-22T20:58:20.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:20.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:20.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705840 2026-02-22T20:58:20.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:20.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705840 2026-02-22T20:58:20.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705840' 2026-02-22T20:58:20.557 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705840 2026-02-22T20:58:20.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:20.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705840 -lt 38654705840 2026-02-22T20:58:20.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:20.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575021 2026-02-22T20:58:20.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:20.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:20.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575021 2026-02-22T20:58:20.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:20.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575021 2026-02-22T20:58:20.718 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575021 2026-02-22T20:58:20.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575021' 2026-02-22T20:58:20.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:20.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575021 -lt 55834575021 2026-02-22T20:58:20.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:20.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:20.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:21.028 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:21.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:21.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:21.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:21.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:21.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:21.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:21.200 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:21.200 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:21.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:21.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:21.200 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:21.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836661 2026-02-22T20:58:21.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836661 2026-02-22T20:58:21.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661' 2026-02-22T20:58:21.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:21.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:21.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705843 2026-02-22T20:58:21.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705843 2026-02-22T20:58:21.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661 1-38654705843' 2026-02-22T20:58:21.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:21.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:21.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575024 2026-02-22T20:58:21.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575024 2026-02-22T20:58:21.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661 1-38654705843 2-55834575024' 2026-02-22T20:58:21.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:21.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836661 2026-02-22T20:58:21.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:21.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:21.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836661 2026-02-22T20:58:21.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:21.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836661 2026-02-22T20:58:21.419 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836661 2026-02-22T20:58:21.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836661' 2026-02-22T20:58:21.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:21.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836659 -lt 21474836661 2026-02-22T20:58:21.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:22.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:22.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:22.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836662 -lt 21474836661 2026-02-22T20:58:22.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:22.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705843 2026-02-22T20:58:22.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:22.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:22.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705843 2026-02-22T20:58:22.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:22.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705843 2026-02-22T20:58:22.748 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705843 2026-02-22T20:58:22.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705843' 2026-02-22T20:58:22.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:22.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705843 -lt 38654705843 2026-02-22T20:58:22.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:22.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575024 2026-02-22T20:58:22.920 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:22.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:22.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575024 2026-02-22T20:58:22.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:22.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575024 2026-02-22T20:58:22.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575024' 2026-02-22T20:58:22.924 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575024 2026-02-22T20:58:22.924 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:23.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575024 -lt 55834575024 2026-02-22T20:58:23.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:23.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:23.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:23.238 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:23.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:23.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:23.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:23.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:23.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:23.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:23.401 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:23.401 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:23.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:23.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:23.401 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:23.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836664 2026-02-22T20:58:23.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836664 2026-02-22T20:58:23.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664' 2026-02-22T20:58:23.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:23.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:23.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705846 2026-02-22T20:58:23.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705846 2026-02-22T20:58:23.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664 1-38654705846' 2026-02-22T20:58:23.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:23.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:23.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575027 2026-02-22T20:58:23.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575027 2026-02-22T20:58:23.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664 1-38654705846 2-55834575027' 2026-02-22T20:58:23.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:23.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836664 2026-02-22T20:58:23.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:23.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:23.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836664 2026-02-22T20:58:23.627 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:23.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836664 2026-02-22T20:58:23.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836664' 2026-02-22T20:58:23.629 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836664 2026-02-22T20:58:23.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:23.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836662 -lt 21474836664 2026-02-22T20:58:23.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:24.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:24.787 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:24.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836665 -lt 21474836664 2026-02-22T20:58:24.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:24.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705846 2026-02-22T20:58:24.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:24.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:24.951 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705846 2026-02-22T20:58:24.951 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:24.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705846 2026-02-22T20:58:24.952 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705846 2026-02-22T20:58:24.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705846' 2026-02-22T20:58:24.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:25.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705846 -lt 38654705846 2026-02-22T20:58:25.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:25.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575027 2026-02-22T20:58:25.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:25.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:25.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575027 2026-02-22T20:58:25.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:25.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575027 2026-02-22T20:58:25.119 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575027 2026-02-22T20:58:25.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575027' 2026-02-22T20:58:25.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:25.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575027 -lt 55834575027 2026-02-22T20:58:25.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:25.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:25.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:25.431 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:25.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:25.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:25.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:25.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:25.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:25.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:25.606 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:25.606 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:25.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:25.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:25.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:25.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836668 2026-02-22T20:58:25.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836668 2026-02-22T20:58:25.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836668' 2026-02-22T20:58:25.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:25.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:25.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705849 2026-02-22T20:58:25.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705849 2026-02-22T20:58:25.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836668 1-38654705849' 2026-02-22T20:58:25.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:25.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:25.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575030 2026-02-22T20:58:25.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575030 2026-02-22T20:58:25.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836668 1-38654705849 2-55834575030' 2026-02-22T20:58:25.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:25.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836668 2026-02-22T20:58:25.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:25.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:25.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836668 2026-02-22T20:58:25.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:25.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836668 2026-02-22T20:58:25.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836668' 2026-02-22T20:58:25.827 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836668 2026-02-22T20:58:25.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:25.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836668 -lt 21474836668 2026-02-22T20:58:25.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:25.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705849 2026-02-22T20:58:25.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:25.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:25.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705849 2026-02-22T20:58:25.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:25.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705849 2026-02-22T20:58:25.997 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705849 2026-02-22T20:58:25.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705849' 2026-02-22T20:58:25.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:26.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705849 -lt 38654705849 2026-02-22T20:58:26.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:26.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575030 2026-02-22T20:58:26.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:26.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:26.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575030 2026-02-22T20:58:26.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:26.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575030 2026-02-22T20:58:26.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575030' 2026-02-22T20:58:26.159 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575030 2026-02-22T20:58:26.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:26.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575029 -lt 55834575030 2026-02-22T20:58:26.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:27.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:27.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:27.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575029 -lt 55834575030 2026-02-22T20:58:27.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:28.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T20:58:28.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:28.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575032 -lt 55834575030 2026-02-22T20:58:28.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:28.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:28.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:28.790 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:28.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:28.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:28.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:28.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:28.801 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:28.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:28.962 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:28.962 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:28.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:28.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:28.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:29.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836672 2026-02-22T20:58:29.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836672 2026-02-22T20:58:29.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836672' 2026-02-22T20:58:29.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:29.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:29.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705854 2026-02-22T20:58:29.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705854 2026-02-22T20:58:29.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836672 1-38654705854' 2026-02-22T20:58:29.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:29.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:29.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575035 2026-02-22T20:58:29.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575035 2026-02-22T20:58:29.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836672 1-38654705854 2-55834575035' 2026-02-22T20:58:29.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:29.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836672 2026-02-22T20:58:29.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:29.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:29.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836672 2026-02-22T20:58:29.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:29.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836672 2026-02-22T20:58:29.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836672' 2026-02-22T20:58:29.183 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836672 2026-02-22T20:58:29.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:29.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836670 -lt 21474836672 2026-02-22T20:58:29.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:30.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:30.353 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:30.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836673 -lt 21474836672 2026-02-22T20:58:30.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:30.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705854 2026-02-22T20:58:30.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:30.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:30.511 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705854 2026-02-22T20:58:30.511 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:30.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705854 2026-02-22T20:58:30.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705854' 2026-02-22T20:58:30.513 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705854 2026-02-22T20:58:30.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:30.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705854 -lt 38654705854 2026-02-22T20:58:30.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:30.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575035 2026-02-22T20:58:30.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:30.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:30.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575035 2026-02-22T20:58:30.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:30.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575035 2026-02-22T20:58:30.673 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575035 2026-02-22T20:58:30.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575035' 2026-02-22T20:58:30.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:30.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575035 -lt 55834575035 2026-02-22T20:58:30.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:30.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:30.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:30.987 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:30.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:30.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:30.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:30.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:30.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:31.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:31.157 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:31.157 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:31.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:31.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:31.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:31.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836675 2026-02-22T20:58:31.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836675 2026-02-22T20:58:31.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836675' 2026-02-22T20:58:31.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:31.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:31.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705857 2026-02-22T20:58:31.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705857 2026-02-22T20:58:31.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836675 1-38654705857' 2026-02-22T20:58:31.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:31.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:31.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575038 2026-02-22T20:58:31.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575038 2026-02-22T20:58:31.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836675 1-38654705857 2-55834575038' 2026-02-22T20:58:31.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:31.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836675 2026-02-22T20:58:31.378 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:31.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:31.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836675 2026-02-22T20:58:31.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:31.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836675 2026-02-22T20:58:31.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836675' 2026-02-22T20:58:31.381 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836675 2026-02-22T20:58:31.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836673 -lt 21474836675 2026-02-22T20:58:31.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:32.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:32.537 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:32.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836676 -lt 21474836675 2026-02-22T20:58:32.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:32.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705857 2026-02-22T20:58:32.696 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:32.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:32.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705857 2026-02-22T20:58:32.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:32.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705857 2026-02-22T20:58:32.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705857' 2026-02-22T20:58:32.700 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705857 2026-02-22T20:58:32.700 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:32.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705857 -lt 38654705857 2026-02-22T20:58:32.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:32.871 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575038 2026-02-22T20:58:32.871 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:32.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:32.873 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575038 2026-02-22T20:58:32.873 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:32.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575038 2026-02-22T20:58:32.874 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575038 2026-02-22T20:58:32.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575038' 2026-02-22T20:58:32.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:33.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575038 -lt 55834575038 2026-02-22T20:58:33.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:33.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:33.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:33.182 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:33.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:33.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:33.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:33.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:33.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:33.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:33.345 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:33.345 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:33.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:33.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:33.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:33.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836678 2026-02-22T20:58:33.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836678 2026-02-22T20:58:33.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678' 2026-02-22T20:58:33.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:33.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:33.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705860 2026-02-22T20:58:33.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705860 2026-02-22T20:58:33.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678 1-38654705860' 2026-02-22T20:58:33.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:33.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:33.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575041 2026-02-22T20:58:33.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575041 2026-02-22T20:58:33.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836678 1-38654705860 2-55834575041' 2026-02-22T20:58:33.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:33.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836678 2026-02-22T20:58:33.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:33.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:33.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836678 2026-02-22T20:58:33.562 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:33.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836678 2026-02-22T20:58:33.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836678' 2026-02-22T20:58:33.563 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836678 2026-02-22T20:58:33.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:33.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836676 -lt 21474836678 2026-02-22T20:58:33.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:34.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:34.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:34.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836679 -lt 21474836678 2026-02-22T20:58:34.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:34.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705860 2026-02-22T20:58:34.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:34.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:34.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705860 2026-02-22T20:58:34.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:34.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705860 2026-02-22T20:58:34.895 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705860 2026-02-22T20:58:34.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705860' 2026-02-22T20:58:34.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:35.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705860 -lt 38654705860 2026-02-22T20:58:35.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:35.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575041 2026-02-22T20:58:35.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:35.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:35.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575041 2026-02-22T20:58:35.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:35.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575041 2026-02-22T20:58:35.064 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575041 2026-02-22T20:58:35.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575041' 2026-02-22T20:58:35.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:35.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575041 -lt 55834575041 2026-02-22T20:58:35.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:35.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:35.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:35.376 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:35.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:35.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:35.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:35.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:35.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:35.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:35.543 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:35.543 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:35.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:35.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:35.544 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:35.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836681 2026-02-22T20:58:35.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836681 2026-02-22T20:58:35.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836681' 2026-02-22T20:58:35.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:35.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:35.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705863 2026-02-22T20:58:35.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705863 2026-02-22T20:58:35.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836681 1-38654705863' 2026-02-22T20:58:35.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:35.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:35.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575044 2026-02-22T20:58:35.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575044 2026-02-22T20:58:35.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836681 1-38654705863 2-55834575044' 2026-02-22T20:58:35.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:35.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836681 2026-02-22T20:58:35.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:35.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:35.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836681 2026-02-22T20:58:35.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:35.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836681 2026-02-22T20:58:35.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836681' 2026-02-22T20:58:35.762 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836681 2026-02-22T20:58:35.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:35.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836682 -lt 21474836681 2026-02-22T20:58:35.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:35.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705863 2026-02-22T20:58:35.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:35.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:35.927 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705863 2026-02-22T20:58:35.927 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:35.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705863 2026-02-22T20:58:35.928 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705863 2026-02-22T20:58:35.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705863' 2026-02-22T20:58:35.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:36.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705863 -lt 38654705863 2026-02-22T20:58:36.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:36.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575044 2026-02-22T20:58:36.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:36.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:36.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575044 2026-02-22T20:58:36.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:36.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575044 2026-02-22T20:58:36.087 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575044 2026-02-22T20:58:36.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575044' 2026-02-22T20:58:36.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:36.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575044 -lt 55834575044 2026-02-22T20:58:36.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:36.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:36.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:36.400 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:36.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:36.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:36.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:36.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:36.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:36.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:36.564 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:36.564 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:36.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:36.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:36.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:36.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836683 2026-02-22T20:58:36.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836683 2026-02-22T20:58:36.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836683' 2026-02-22T20:58:36.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:36.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:36.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705865 2026-02-22T20:58:36.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705865 2026-02-22T20:58:36.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836683 1-38654705865' 2026-02-22T20:58:36.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:36.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:36.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575046 2026-02-22T20:58:36.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575046 2026-02-22T20:58:36.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836683 1-38654705865 2-55834575046' 2026-02-22T20:58:36.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:36.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836683 2026-02-22T20:58:36.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:36.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:36.782 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836683 2026-02-22T20:58:36.782 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:36.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836683 2026-02-22T20:58:36.783 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836683 2026-02-22T20:58:36.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836683' 2026-02-22T20:58:36.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:36.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836682 -lt 21474836683 2026-02-22T20:58:36.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:37.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:37.952 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:38.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836685 -lt 21474836683 2026-02-22T20:58:38.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:38.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705865 2026-02-22T20:58:38.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:38.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:38.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705865 2026-02-22T20:58:38.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:38.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705865 2026-02-22T20:58:38.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705865' 2026-02-22T20:58:38.117 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705865 2026-02-22T20:58:38.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:38.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705866 -lt 38654705865 2026-02-22T20:58:38.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:38.282 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575046 2026-02-22T20:58:38.282 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:38.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:38.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575046 2026-02-22T20:58:38.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:38.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575046 2026-02-22T20:58:38.286 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575046 2026-02-22T20:58:38.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575046' 2026-02-22T20:58:38.286 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:38.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575047 -lt 55834575046 2026-02-22T20:58:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:38.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:38.588 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:38.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:38.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:38.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:38.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:38.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:38.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:38.756 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:38.756 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:38.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:38.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:38.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:38.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836687 2026-02-22T20:58:38.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836687 2026-02-22T20:58:38.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836687' 2026-02-22T20:58:38.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:38.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:38.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705868 2026-02-22T20:58:38.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705868 2026-02-22T20:58:38.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836687 1-38654705868' 2026-02-22T20:58:38.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:38.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:38.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575050 2026-02-22T20:58:38.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575050 2026-02-22T20:58:38.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836687 1-38654705868 2-55834575050' 2026-02-22T20:58:38.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:38.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836687 2026-02-22T20:58:38.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:38.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:38.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836687 2026-02-22T20:58:38.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:38.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836687 2026-02-22T20:58:38.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836687' 2026-02-22T20:58:38.978 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836687 2026-02-22T20:58:38.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:39.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836685 -lt 21474836687 2026-02-22T20:58:39.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:40.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:40.145 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:40.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836688 -lt 21474836687 2026-02-22T20:58:40.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:40.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705868 2026-02-22T20:58:40.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:40.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:40.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705868 2026-02-22T20:58:40.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:40.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705868 2026-02-22T20:58:40.308 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705868 2026-02-22T20:58:40.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705868' 2026-02-22T20:58:40.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:40.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705869 -lt 38654705868 2026-02-22T20:58:40.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:40.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575050 2026-02-22T20:58:40.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:40.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:40.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575050 2026-02-22T20:58:40.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:40.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575050 2026-02-22T20:58:40.477 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575050 2026-02-22T20:58:40.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575050' 2026-02-22T20:58:40.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:40.634 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575050 -lt 55834575050 2026-02-22T20:58:40.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:40.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:40.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:40.782 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:40.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:40.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:40.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:40.792 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:40.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:40.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:40.951 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:40.951 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:40.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:40.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:40.951 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:41.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836690 2026-02-22T20:58:41.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836690 2026-02-22T20:58:41.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836690' 2026-02-22T20:58:41.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:41.024 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:41.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705872 2026-02-22T20:58:41.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705872 2026-02-22T20:58:41.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836690 1-38654705872' 2026-02-22T20:58:41.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:41.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:41.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575053 2026-02-22T20:58:41.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575053 2026-02-22T20:58:41.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836690 1-38654705872 2-55834575053' 2026-02-22T20:58:41.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:41.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836690 2026-02-22T20:58:41.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:41.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:41.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836690 2026-02-22T20:58:41.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:41.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836690 2026-02-22T20:58:41.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836690' 2026-02-22T20:58:41.173 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836690 2026-02-22T20:58:41.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:41.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836688 -lt 21474836690 2026-02-22T20:58:41.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:42.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:42.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:42.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836691 -lt 21474836690 2026-02-22T20:58:42.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:42.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705872 2026-02-22T20:58:42.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:42.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:42.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705872 2026-02-22T20:58:42.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:42.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705872 2026-02-22T20:58:42.495 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705872 2026-02-22T20:58:42.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705872' 2026-02-22T20:58:42.495 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:42.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705872 -lt 38654705872 2026-02-22T20:58:42.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:42.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575053 2026-02-22T20:58:42.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:42.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:42.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575053 2026-02-22T20:58:42.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:42.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575053 2026-02-22T20:58:42.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575053' 2026-02-22T20:58:42.671 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575053 2026-02-22T20:58:42.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:42.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575053 -lt 55834575053 2026-02-22T20:58:42.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:42.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:42.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:42.985 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:42.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T20:58:42.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T20:58:42.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T20:58:42.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T20:58:42.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T20:58:43.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T20:58:43.149 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T20:58:43.149 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T20:58:43.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T20:58:43.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:43.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T20:58:43.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836693 2026-02-22T20:58:43.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836693 2026-02-22T20:58:43.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836693' 2026-02-22T20:58:43.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:43.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T20:58:43.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705875 2026-02-22T20:58:43.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705875 2026-02-22T20:58:43.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836693 1-38654705875' 2026-02-22T20:58:43.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T20:58:43.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T20:58:43.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834575056 2026-02-22T20:58:43.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834575056 2026-02-22T20:58:43.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836693 1-38654705875 2-55834575056' 2026-02-22T20:58:43.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:43.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836693 2026-02-22T20:58:43.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:43.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T20:58:43.374 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836693 2026-02-22T20:58:43.374 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:43.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836693 2026-02-22T20:58:43.376 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836693 2026-02-22T20:58:43.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836693' 2026-02-22T20:58:43.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:43.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836691 -lt 21474836693 2026-02-22T20:58:43.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T20:58:44.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T20:58:44.542 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T20:58:44.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836694 -lt 21474836693 2026-02-22T20:58:44.703 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:44.704 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705875 2026-02-22T20:58:44.704 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:44.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T20:58:44.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705875 2026-02-22T20:58:44.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:44.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705875 2026-02-22T20:58:44.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705875' 2026-02-22T20:58:44.707 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705875 2026-02-22T20:58:44.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T20:58:44.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705875 -lt 38654705875 2026-02-22T20:58:44.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T20:58:44.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834575056 2026-02-22T20:58:44.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T20:58:44.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T20:58:44.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-55834575056 2026-02-22T20:58:44.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T20:58:44.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=55834575056 2026-02-22T20:58:44.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 55834575056' 2026-02-22T20:58:44.878 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 55834575056 2026-02-22T20:58:44.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T20:58:45.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 55834575056 -lt 55834575056 2026-02-22T20:58:45.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T20:58:45.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T20:58:45.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T20:58:45.193 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T20:58:45.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: break 2026-02-22T20:58:45.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: set +o pipefail 2026-02-22T20:58:45.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: sleep 5 2026-02-22T20:58:50.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:384: _scrub_abort: grep 'nodeep_scrub set, aborting' td/osd-scrub-test/osd.1.log 2026-02-22T20:58:50.211 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T20:56:48.056+0000 7ff50eda9640 10 osd.1 pg_epoch: 18 pg[1.0( v 16'1000 (0'0,16'1000] local-lis/les=14/15 n=1000 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0,2] r=0 lpr=14 crt=16'1000 lcod 16'999 mlcod 16'999 active+clean+scrubbing+deep(periodic-regular)] scrubber: nodeep_scrub set, aborting 2026-02-22T20:58:50.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:390: _scrub_abort: get_last_scrub_stamp 1.0 2026-02-22T20:58:50.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:58:50.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:58:50.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:58:50.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:58:50.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:390: _scrub_abort: local last_scrub=2026-02-22T20:56:13.150001+0000 2026-02-22T20:58:50.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:391: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2026-02-22T20:58:50.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:393: _scrub_abort: ceph osd unset nodeep-scrub 2026-02-22T20:58:50.726 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is unset 2026-02-22T20:58:50.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:394: _scrub_abort: '[' deep-scrub = deep-scrub ']' 2026-02-22T20:58:50.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:396: _scrub_abort: ceph osd unset noscrub 2026-02-22T20:58:50.927 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is unset 2026-02-22T20:58:50.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: TIMEOUT=500 2026-02-22T20:58:50.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:399: _scrub_abort: wait_for_scrub 1.0 2026-02-22T20:56:13.150001+0000 2026-02-22T20:58:50.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T20:58:50.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T20:56:13.150001+0000 2026-02-22T20:58:50.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T20:58:50.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T20:58:50.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:58:50.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:58:50.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:58:50.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:58:50.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:58:50.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:58:51.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:58:51.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:58:52.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:58:52.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:58:52.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:58:52.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:58:52.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:58:52.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:58:52.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:58:52.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:58:52.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:58:53.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:58:53.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:58:53.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:58:53.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:58:53.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:58:53.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:58:53.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:58:53.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:58:53.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:58:54.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:58:54.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:58:54.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:58:54.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:58:54.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:58:54.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:58:54.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:58:54.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:58:54.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:58:55.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:58:55.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:58:55.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:58:55.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:58:55.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:58:55.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:58:55.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:58:55.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:58:55.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:58:56.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:58:56.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:58:56.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:58:56.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:58:56.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:58:56.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:58:56.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:58:56.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:58:56.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:58:57.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:58:57.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:58:57.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:58:57.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:58:57.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:58:57.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:58:57.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:58:58.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:58:58.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:58:59.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:58:59.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:58:59.069 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:58:59.069 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:58:59.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:58:59.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:58:59.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:58:59.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:58:59.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:00.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:00.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:00.224 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:00.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:00.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:00.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:00.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:00.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:00.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:01.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:01.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:01.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:01.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:01.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:01.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:01.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:01.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:01.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:02.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:02.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:02.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:02.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:02.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:02.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:02.546 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:02.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:02.700 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:03.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:03.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:03.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:03.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:03.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:03.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:03.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:03.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:03.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:04.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:04.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:04.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:04.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:04.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:04.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:04.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:05.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:05.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:06.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:06.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:06.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:06.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:06.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:06.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:06.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:06.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:06.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:07.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:07.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:07.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:07.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:07.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:07.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:07.172 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:07.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:07.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:08.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:08.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:08.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:08.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:08.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:08.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:08.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:08.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:08.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:09.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:09.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:09.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:09.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:09.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:09.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:09.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:09.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:09.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:10.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:10.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:10.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:10.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:10.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:10.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:10.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:10.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:10.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:11.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:11.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:11.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:11.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:11.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:11.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:11.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:11.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:11.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:12.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:12.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:12.976 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:12.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:12.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:12.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:12.977 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:13.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:13.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:14.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:14.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:14.133 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:14.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:14.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:14.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:14.134 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:14.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:14.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:15.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:15.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:15.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:15.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:15.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:15.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:15.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:15.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:15.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:16.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:16.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:16.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:16.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:16.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:16.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:16.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:16.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:16.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:17.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:17.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:17.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:17.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:17.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:17.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:17.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:17.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:17.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:18.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:18.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:18.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:18.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:18.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:18.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:18.773 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:18.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:18.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:19.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:19.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:19.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:19.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:19.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:19.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:19.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:20.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:20.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:21.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:21.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:21.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:21.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:21.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:21.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:21.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:21.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:21.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:22.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:22.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:22.254 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:22.254 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:22.254 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:22.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:22.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:22.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:22.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:23.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:23.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:23.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:23.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:23.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:23.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:23.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:23.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:23.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:24.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:24.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:24.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:24.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:24.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:24.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:24.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:24.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:24.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:25.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:25.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:25.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:25.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:25.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:25.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:25.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:25.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:25.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:26.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:26.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:26.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:26.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:26.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:26.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:26.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:27.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:27.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:28.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:28.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:28.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:28.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:28.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:28.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:28.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:28.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:28.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:29.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:29.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:29.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:29.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:29.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:29.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:29.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:29.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:29.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:30.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:30.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:30.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:30.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:30.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:30.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:30.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:30.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:30.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:31.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:31.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:31.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:31.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:31.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:31.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:31.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:31.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:31.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:32.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:32.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:32.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:32.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:32.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:32.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:32.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:32.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:32.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:33.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:33.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:33.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:33.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:33.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:33.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:33.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:34.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:34.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:35.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:35.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:35.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:35.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:35.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:35.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:35.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:35.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:35.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:36.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:36.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:36.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:36.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:36.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:36.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:36.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:36.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:36.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:37.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:37.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:37.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:37.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:37.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:37.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:37.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:37.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:37.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:38.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:38.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:38.473 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:38.473 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:38.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:38.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:38.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:38.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:38.626 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:39.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:39.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:39.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:39.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:39.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:39.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:39.629 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:39.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:39.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:40.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:40.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:40.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:40.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:40.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:40.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:40.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:40.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:40.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:41.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:41.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:41.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:41.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:41.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:41.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:41.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:42.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:42.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:43.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:43.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:43.107 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:43.107 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:43.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:43.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:43.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:43.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:43.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:44.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:44.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:44.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:44.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:44.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:44.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:44.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:44.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:44.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:45.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:45.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:45.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:45.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:45.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:45.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:45.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:45.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:45.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:46.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:46.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:46.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:46.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:46.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:46.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:46.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:46.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:46.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:47.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:47.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:47.750 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:47.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:47.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:47.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:47.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:47.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:47.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:48.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:48.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:48.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:48.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:48.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:48.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:48.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:49.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:49.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:50.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:50.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:50.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:50.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:50.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:50.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:50.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:50.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:50.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:51.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:51.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:51.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:51.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:51.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:51.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:51.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:51.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:51.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:52.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:52.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:52.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:52.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:52.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:52.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:52.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:52.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:52.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:53.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:53.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:53.556 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:53.556 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:53.556 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:53.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:53.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:53.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:53.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:54.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:54.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:54.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:54.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:54.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:54.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:54.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:54.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:54.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:55.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:55.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:55.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:55.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:55.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:55.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:55.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:56.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:56.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:57.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:57.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:57.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:57.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:57.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:57.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:57.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:57.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:57.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:58.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:58.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:58.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:58.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:58.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:58.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:58.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:58.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:58.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T20:59:59.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T20:59:59.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T20:59:59.348 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T20:59:59.348 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T20:59:59.348 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T20:59:59.349 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T20:59:59.349 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T20:59:59.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T20:59:59.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:00.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:00.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:00.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:00.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:00.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:00.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:00.506 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:00.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:00.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:01.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:01.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:01.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:01.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:01.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:01.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:01.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:01.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:01.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:02.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:02.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:02.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:02.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:02.821 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:02.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:02.822 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:02.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:02.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:03.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:03.977 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:03.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:03.978 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:03.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:03.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:03.979 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:04.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:04.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:05.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:05.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:05.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:05.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:05.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:05.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:05.138 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:05.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:05.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:06.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:06.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:06.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:06.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:06.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:06.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:06.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:06.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:06.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:07.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:07.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:07.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:07.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:07.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:07.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:07.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:07.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:07.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:08.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:08.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:08.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:08.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:08.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:08.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:08.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:08.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:08.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:09.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:09.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:09.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:09.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:09.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:09.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:09.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:09.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:09.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:10.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:10.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:10.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:10.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:10.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:10.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:10.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:11.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:11.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:12.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:12.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:12.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:12.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:12.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:12.093 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:12.093 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:12.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:12.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:13.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:13.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:13.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:13.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:13.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:13.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:13.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:13.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:13.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:14.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:14.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:14.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:14.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:14.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:14.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:14.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:14.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:14.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:15.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:15.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:15.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:15.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:15.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:15.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:15.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:15.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:15.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:16.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:16.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:16.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:16.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:16.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:16.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:16.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:16.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:16.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:17.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:17.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:17.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:17.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:17.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:17.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:17.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:18.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:18.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:19.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:19.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:19.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:19.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:19.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:19.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:19.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:19.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:19.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:20.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:20.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:20.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:20.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:20.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:20.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:20.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:20.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:20.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:21.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:21.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:21.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:21.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:21.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:21.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:21.368 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:21.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:21.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:22.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:22.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:22.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:22.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:22.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:22.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:22.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:22.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:22.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:23.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:23.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:23.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:23.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:23.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:23.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:23.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:23.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:23.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:24.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:24.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:24.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:24.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:24.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:24.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:24.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:25.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:25.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:26.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:26.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:26.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:26.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:26.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:26.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:26.007 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:26.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:26.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:27.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:27.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:27.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:27.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:27.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:27.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:27.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:27.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:27.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:28.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:28.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:28.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:28.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:28.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:28.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:28.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:28.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:28.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:29.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:29.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:29.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:29.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:29.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:29.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:29.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:29.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:29.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:30.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:30.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:30.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:30.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:30.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:30.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:30.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:30.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:30.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:31.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:31.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:31.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:31.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:31.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:31.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:31.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:31.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:31.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:32.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:32.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:32.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:32.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:32.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:32.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:32.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:33.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:33.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:34.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:34.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:34.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:34.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:34.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:34.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:34.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:34.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:34.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:35.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:35.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:35.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:35.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:35.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:35.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:35.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:35.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:35.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:36.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:36.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:36.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:36.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:36.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:36.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:36.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:36.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:36.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:37.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:37.602 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:37.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:37.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:37.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:37.603 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:37.603 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:37.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:37.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:38.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:38.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:38.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:38.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:38.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:38.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:38.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:38.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:38.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:39.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:39.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:39.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:39.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:39.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:39.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:39.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:40.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:40.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:41.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:41.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:41.088 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:41.088 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:41.088 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:41.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:41.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:41.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:41.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:42.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:42.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:42.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:42.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:42.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:42.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:42.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:42.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:42.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:43.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:43.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:43.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:43.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:43.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:43.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:43.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:43.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:43.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:44.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:44.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:44.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:44.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:44.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:44.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:44.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:44.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:44.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:45.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:45.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:45.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:45.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:45.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:45.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:45.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:45.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:45.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:46.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:46.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:46.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:46.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:46.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:46.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:46.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:47.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:47.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:48.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:48.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:48.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:48.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:48.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:48.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:48.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:48.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:48.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:49.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:49.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:49.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:49.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:49.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:49.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:49.198 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:49.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:49.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:50.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:50.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:50.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:50.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:50.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:50.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:50.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:50.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:50.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:51.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:51.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:51.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:51.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:51.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:51.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:51.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:51.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:51.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:52.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:52.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:52.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:52.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:52.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:52.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:52.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:52.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:52.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:53.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:53.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:53.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:53.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:53.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:53.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:53.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:53.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:53.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:54.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:54.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:54.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:54.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:54.994 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:54.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:54.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:55.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:55.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:56.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:56.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:56.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:56.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:56.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:56.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:56.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:56.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:56.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:57.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:57.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:57.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:57.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:57.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:57.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:57.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:57.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:57.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:58.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:58.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:58.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:58.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:58.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:58.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:58.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:58.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:58.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:00:59.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:00:59.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:00:59.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:00:59.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:00:59.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:00:59.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:00:59.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:00:59.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:00:59.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:00.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:00.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:00.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:00.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:00.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:00.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:00.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:00.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:00.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:01.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:01.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:01.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:01.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:01.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:01.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:01.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:02.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:02.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:03.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:03.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:03.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:03.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:03.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:03.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:03.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:03.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:03.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:04.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:04.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:04.282 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:04.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:04.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:04.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:04.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:04.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:04.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:05.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:05.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:05.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:05.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:05.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:05.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:05.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:05.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:05.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:06.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:06.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:06.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:06.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:06.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:06.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:06.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:06.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:06.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:07.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:07.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:07.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:07.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:07.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:07.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:07.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:07.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:07.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:08.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:08.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:08.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:08.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:08.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:08.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:08.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:09.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:09.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:10.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:10.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:10.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:10.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:10.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:10.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:10.073 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:10.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:10.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:11.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:11.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:11.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:11.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:11.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:11.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:11.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:11.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:11.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:12.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:12.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:12.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:12.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:12.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:12.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:12.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:12.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:12.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:13.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:13.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:13.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:13.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:13.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:13.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:13.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:13.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:13.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:14.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:14.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:14.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:14.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:14.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:14.720 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:14.720 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:14.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:14.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:15.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:15.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:15.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:15.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:15.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:15.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:15.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:16.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:16.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:17.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:17.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:17.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:17.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:17.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:17.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:17.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:17.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:17.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:18.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:18.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:18.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:18.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:18.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:18.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:18.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:18.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:18.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:19.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:19.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:19.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:19.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:19.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:19.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:19.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:19.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:19.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:20.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:20.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:20.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:20.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:20.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:20.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:20.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:20.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:20.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:21.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:21.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:21.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:21.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:21.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:21.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:21.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:21.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:21.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:22.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:22.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:22.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:22.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:22.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:22.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:22.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:22.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:22.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:23.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:23.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:23.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:23.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:23.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:23.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:23.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:24.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:24.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:25.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:25.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:25.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:25.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:25.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:25.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:25.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:25.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:25.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:26.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:26.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:26.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:26.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:26.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:26.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:26.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:26.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:26.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:27.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:27.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:27.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:27.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:27.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:27.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:27.471 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:27.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:27.630 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:28.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:28.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:28.632 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:28.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:28.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:28.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:28.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:28.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:28.790 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:29.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:29.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:29.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:29.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:29.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:29.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:29.792 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:29.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:29.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:30.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:30.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:30.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:30.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:30.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:30.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:30.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:31.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:31.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:32.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:32.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:32.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:32.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:32.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:32.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:32.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:32.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:32.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:33.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:33.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:33.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:33.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:33.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:33.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:33.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:33.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:33.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:34.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:34.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:34.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:34.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:34.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:34.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:34.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:34.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:34.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:35.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:35.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:35.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:35.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:35.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:35.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:35.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:35.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:35.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:36.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:36.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:36.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:36.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:36.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:36.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:36.760 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:36.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:36.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:37.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:37.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:37.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:37.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:37.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:37.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:37.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:38.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:38.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:39.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:39.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:39.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:39.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:39.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:39.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:39.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:39.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:39.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:40.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:40.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:40.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:40.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:40.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:40.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:40.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:40.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:40.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:41.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:41.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:41.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:41.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:41.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:41.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:41.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:41.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:41.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:42.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:42.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:42.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:42.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:42.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:42.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:42.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:42.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:42.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:43.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:43.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:43.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:43.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:43.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:43.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:43.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:43.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:43.865 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:44.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:44.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:44.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:44.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:44.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:44.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:44.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:45.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:45.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:46.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:46.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:46.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:46.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:46.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:46.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:46.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:46.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:46.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:47.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:47.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:47.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:47.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:47.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:47.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:47.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:47.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:47.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:48.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:48.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:48.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:48.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:48.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:48.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:48.346 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:48.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:48.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:49.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:49.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:49.504 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:49.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:49.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:49.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:49.505 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:49.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:49.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:50.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:50.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:50.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:50.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:50.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:50.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:50.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:50.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:50.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:51.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:51.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:51.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:51.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:51.824 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:51.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:51.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:51.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:51.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:52.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:52.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:52.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:52.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:52.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:52.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:52.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:53.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:53.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:54.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:54.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:54.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:54.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:54.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:54.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:54.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:54.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:54.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:55.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:55.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:55.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:55.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:55.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:55.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:55.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:55.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:55.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:56.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:56.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:56.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:56.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:56.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:56.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:56.452 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:56.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:56.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:57.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:57.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:57.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:57.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:57.606 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:57.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:57.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:57.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:57.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:58.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:58.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:58.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:58.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:58.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:58.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:58.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:01:58.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:01:58.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:01:59.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:01:59.929 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:01:59.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:01:59.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:01:59.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:01:59.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:01:59.930 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:00.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:00.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:01.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:01.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:01.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:01.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:01.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:01.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:01.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:01.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:01.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:02.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:02.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:02.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:02.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:02.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:02.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:02.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:02.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:02.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:03.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:03.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:03.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:03.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:03.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:03.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:03.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:03.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:03.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:04.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:04.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:04.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:04.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:04.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:04.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:04.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:04.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:04.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:05.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:05.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:05.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:05.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:05.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:05.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:05.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:05.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:05.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:06.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:06.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:06.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:06.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:06.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:06.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:06.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:07.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:07.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:08.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:08.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:08.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:08.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:08.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:08.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:08.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:08.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:08.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:09.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:09.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:09.202 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:09.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:09.203 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:09.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:09.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:09.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:09.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:10.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:10.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:10.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:10.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:10.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:10.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:10.359 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:10.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:10.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:11.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:11.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:11.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:11.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:11.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:11.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:11.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:11.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:11.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:12.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:12.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:12.680 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:12.680 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:12.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:12.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:12.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:12.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:12.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:13.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:13.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:13.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:13.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:13.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:13.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:13.842 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:14.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:14.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:15.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:15.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:15.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:15.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:15.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:15.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:15.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:15.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:15.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:16.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:16.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:16.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:16.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:16.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:16.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:16.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:16.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:16.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:17.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:17.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:17.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:17.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:17.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:17.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:17.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:17.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:17.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:18.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:18.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:18.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:18.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:18.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:18.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:18.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:18.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:18.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:19.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:19.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:19.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:19.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:19.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:19.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:19.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:19.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:19.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:20.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:20.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:20.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:20.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:20.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:20.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:20.795 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:20.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:20.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:21.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:21.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:21.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:21.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:21.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:21.953 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:21.954 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:22.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:22.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:23.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:23.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:23.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:23.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:23.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:23.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:23.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:23.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:23.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:24.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:24.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:24.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:24.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:24.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:24.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:24.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:24.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:24.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:25.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:25.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:25.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:25.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:25.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:25.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:25.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:25.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:25.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:26.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:26.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:26.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:26.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:26.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:26.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:26.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:26.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:26.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:27.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:27.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:27.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:27.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:27.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:27.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:27.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:27.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:27.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:28.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:28.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:28.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:28.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:28.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:28.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:28.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:29.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:29.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:30.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:30.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:30.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:30.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:30.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:30.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:30.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:30.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:30.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:31.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:31.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:31.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:31.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:31.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:31.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:31.213 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:31.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:31.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:32.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:32.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:32.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:32.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:32.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:32.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:32.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:32.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:32.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:33.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:33.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:33.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:33.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:33.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:33.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:33.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:33.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:33.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:34.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:34.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:34.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:34.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:34.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:34.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:34.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:34.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:34.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:35.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:35.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:35.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:35.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:35.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:35.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:35.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:36.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:36.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:37.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:37.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:37.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:37.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:37.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:37.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:37.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:37.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:37.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:38.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:38.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:38.167 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:38.167 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:38.167 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:38.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:38.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:38.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:38.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:39.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:39.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:39.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:39.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:39.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:39.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:39.327 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:39.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:39.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:40.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:40.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:40.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:40.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:40.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:40.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:40.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:40.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:40.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:41.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:41.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:41.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:41.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:41.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:41.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:41.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:41.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:41.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:42.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:42.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:42.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:42.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:42.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:42.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:42.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:42.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:42.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:43.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:43.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:43.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:43.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:43.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:43.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:43.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:44.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:44.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:45.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:45.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:45.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:45.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:45.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:45.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:45.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:45.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:45.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:46.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:46.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:46.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:46.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:46.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:46.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:46.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:46.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:46.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:47.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:47.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:47.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:47.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:47.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:47.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:47.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:47.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:47.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:48.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:48.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:48.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:48.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:48.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:48.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:48.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:48.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:48.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:49.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:49.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:49.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:49.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:49.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:49.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:49.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:49.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:49.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:50.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:50.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:50.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:50.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:50.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:50.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:50.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:51.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:51.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:52.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:52.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:52.100 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:52.100 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:52.100 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:52.100 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:52.100 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:52.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:52.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:53.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:53.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:53.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:53.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:53.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:53.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:53.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:53.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:53.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:54.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:54.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:54.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:54.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:54.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:54.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:54.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:54.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:54.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:55.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:55.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:55.574 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:55.574 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:55.574 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:55.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:55.575 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:55.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:55.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:56.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:56.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:56.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:56.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:56.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:56.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:56.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:56.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:56.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:57.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:57.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:57.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:57.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:57.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:57.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:57.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:58.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:58.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:02:59.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:02:59.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:02:59.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:02:59.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:02:59.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:02:59.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:02:59.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:02:59.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:02:59.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:00.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:00.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:00.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:00.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:00.213 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:00.213 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:00.213 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:00.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:00.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:01.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:01.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:01.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:01.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:01.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:01.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:01.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:01.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:01.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:02.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:02.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:02.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:02.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:02.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:02.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:02.529 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:02.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:02.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:03.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:03.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:03.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:03.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:03.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:03.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:03.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:03.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:03.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:04.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:04.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:04.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:04.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:04.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:04.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:04.838 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:04.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:04.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:05.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:05.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:05.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:05.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:05.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:05.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:05.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:06.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:06.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:07.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:07.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:07.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:07.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:07.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:07.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:07.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:07.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:07.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:08.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:08.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:08.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:08.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:08.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:08.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:08.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:08.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:08.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:09.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:09.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:09.465 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:09.466 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:09.466 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:09.466 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:09.466 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:09.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:09.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:10.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:10.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:10.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:10.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:10.625 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:10.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:10.626 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:10.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:10.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:11.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:11.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:11.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:11.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:11.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:11.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:11.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:11.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:11.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:12.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:12.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:12.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:12.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:12.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:12.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:12.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:13.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:13.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:14.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:14.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:14.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:14.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:14.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:14.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:14.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:14.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:14.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:15.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:15.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:15.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:15.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:15.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:15.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:15.260 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:15.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:15.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:16.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:16.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:16.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:16.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:16.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:16.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:16.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:16.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:16.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:17.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:17.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:17.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:17.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:17.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:17.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:17.572 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:17.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:17.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:18.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:18.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:18.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:18.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:18.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:18.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:18.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:18.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:18.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:19.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:19.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:19.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:19.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:19.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:19.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:19.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:20.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:20.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:21.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:21.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:21.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:21.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:21.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:21.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:21.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:21.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:21.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:22.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:22.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:22.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:22.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:22.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:22.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:22.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:22.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:22.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:23.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:23.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:23.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:23.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:23.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:23.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:23.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:23.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:23.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:24.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:24.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:24.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:24.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:24.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:24.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:24.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:24.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:24.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:25.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:25.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:25.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:25.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:25.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:25.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:25.670 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:25.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:25.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:26.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:26.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:26.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:26.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:26.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:26.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:26.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:26.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:26.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:27.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:27.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:27.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:27.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:27.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:27.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:27.997 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:28.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:28.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:29.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:29.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:29.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:29.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:29.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:29.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:29.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:29.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:29.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:30.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:30.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:30.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:30.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:30.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:30.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:30.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:30.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:30.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:31.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:31.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:31.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:31.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:31.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:31.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:31.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:31.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:31.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:32.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:32.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:32.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:32.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:32.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:32.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:32.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:32.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:32.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:33.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:33.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:33.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:33.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:33.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:33.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:33.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:33.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:33.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:34.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:34.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:34.957 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:34.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:34.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:34.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:34.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:35.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:35.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:36.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:36.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:36.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:36.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:36.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:36.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:36.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:36.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:36.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:37.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:37.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:37.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:37.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:37.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:37.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:37.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:37.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:37.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:38.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:38.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:38.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:38.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:38.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:38.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:38.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:38.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:38.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:39.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:39.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:39.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:39.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:39.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:39.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:39.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:39.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:39.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:40.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:40.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:40.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:40.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:40.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:40.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:40.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:40.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:40.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:41.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:41.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:41.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:41.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:41.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:41.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:41.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:42.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:42.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:43.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:43.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:43.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:43.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:43.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:43.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:43.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:43.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:43.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:44.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:44.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:44.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:44.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:44.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:44.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:44.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:44.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:44.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:45.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:45.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:45.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:45.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:45.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:45.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:45.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:45.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:45.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:46.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:46.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:46.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:46.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:46.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:46.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:46.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:46.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:46.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:47.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:47.676 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:47.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:47.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:47.676 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:47.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:47.677 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:47.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:47.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:48.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:48.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:48.830 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:48.830 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:48.830 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:48.830 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:48.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:48.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:48.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:49.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:49.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:49.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:49.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:49.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:49.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:49.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:50.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:50.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:51.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:51.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:51.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:51.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:51.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:51.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:51.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:51.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:51.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:52.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:52.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:52.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:52.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:52.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:52.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:52.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:52.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:52.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:53.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:53.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:53.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:53.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:53.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:53.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:53.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:53.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:53.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:54.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:54.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:54.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:54.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:54.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:54.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:54.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:54.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:54.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:55.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:55.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:55.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:55.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:55.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:55.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:55.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:55.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:55.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:56.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:56.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:56.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:56.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:56.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:56.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:56.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:57.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:57.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:58.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:58.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:58.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:58.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:58.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:58.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:58.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:58.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:58.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:03:59.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:03:59.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:03:59.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:03:59.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:03:59.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:03:59.286 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:03:59.286 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:03:59.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:03:59.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:00.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:00.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:00.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:00.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:00.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:00.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:00.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:00.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:00.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:01.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:01.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:01.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:01.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:01.604 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:01.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:01.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:01.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:01.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:02.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:02.766 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:02.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:02.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:02.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:02.767 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:02.767 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:02.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:02.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:03.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:03.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:03.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:03.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:03.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:03.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:03.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:04.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:04.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:05.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:05.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:05.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:05.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:05.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:05.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:05.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:05.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:05.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:06.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:06.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:06.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:06.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:06.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:06.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:06.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:06.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:06.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:07.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:07.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:07.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:07.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:07.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:07.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:07.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:07.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:07.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:08.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:08.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:08.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:08.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:08.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:08.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:08.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:08.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:08.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:09.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:09.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:09.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:09.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:09.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:09.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:09.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:09.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:09.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:10.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:10.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:10.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:10.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:10.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:10.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:10.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:11.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:11.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:12.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:12.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:12.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:12.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:12.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:12.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:12.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:12.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:12.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:13.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:13.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:13.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:13.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:13.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:13.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:13.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:13.354 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:13.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:14.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:14.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:14.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:14.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:14.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:14.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:14.356 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:14.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:14.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:15.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:15.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:15.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:15.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:15.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:15.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:15.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:15.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:15.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:16.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:16.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:16.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:16.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:16.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:16.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:16.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:16.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:16.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:17.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:17.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:17.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:17.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:17.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:17.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:17.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:17.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:17.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:18.979 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:18.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:18.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:18.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:18.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:18.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:18.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:19.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:19.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:20.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:20.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:20.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:20.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:20.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:20.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:20.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:20.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:21.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:21.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:21.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:21.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:21.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:21.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:21.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:21.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:21.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:22.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:22.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:22.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:22.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:22.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:22.456 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:22.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:22.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:22.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:23.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:23.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:23.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:23.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:23.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:23.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:23.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:23.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:23.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:24.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:24.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:24.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:24.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:24.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:24.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:24.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:24.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:24.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:25.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:25.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:25.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:25.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:25.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:25.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:25.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:26.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:26.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:27.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:27.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:27.100 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:27.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:27.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:27.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:27.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:27.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:27.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:28.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:28.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:28.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:28.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:28.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:28.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:28.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:28.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:28.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:29.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:29.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:29.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:29.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:29.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:29.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:29.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:29.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:29.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:30.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:30.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:30.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:30.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:30.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:30.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:30.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:30.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:30.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:31.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:31.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:31.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:31.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:31.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:31.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:31.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:31.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:31.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:32.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:32.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:32.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:32.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:32.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:32.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:32.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:33.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:33.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:34.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:34.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:34.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:34.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:34.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:34.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:34.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:34.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:34.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:35.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:35.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:35.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:35.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:35.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:35.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:35.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:35.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:35.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:36.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:36.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:36.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:36.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:36.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:36.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:36.374 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:36.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:36.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:37.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:37.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:37.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:37.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:37.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:37.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:37.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:37.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:37.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:38.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:38.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:38.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:38.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:38.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:38.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:38.682 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:38.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:38.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:39.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:39.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:39.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:39.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:39.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:39.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:39.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:39.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:39.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:40.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:40.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:40.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:40.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:40.990 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:40.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:40.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:41.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:41.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:42.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:42.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:42.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:42.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:42.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:42.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:42.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:42.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:42.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:43.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:43.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:43.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:43.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:43.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:43.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:43.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:43.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:43.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:44.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:44.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:44.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:44.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:44.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:44.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:44.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:44.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:44.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:45.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:45.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:45.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:45.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:45.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:45.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:45.624 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:45.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:45.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:46.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:46.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:46.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:46.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:46.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:46.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:46.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:46.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:46.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:47.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:47.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:47.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:47.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:47.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:47.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:47.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:48.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:48.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:49.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:49.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:49.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:49.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:49.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:49.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:49.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:49.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:49.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:50.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:50.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:50.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:50.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:50.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:50.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:50.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:50.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:50.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:51.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:51.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:51.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:51.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:51.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:51.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:51.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:51.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:51.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:52.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:52.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:52.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:52.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:52.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:52.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:52.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:52.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:52.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:53.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:53.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:53.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:53.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:53.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:53.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:53.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:53.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:53.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:54.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:54.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:54.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:54.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:54.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:54.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:54.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:55.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:55.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:56.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:56.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:56.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:56.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:56.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:56.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:56.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:56.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:56.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:57.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:57.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:57.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:57.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:57.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:57.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:57.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:57.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T20:56:13.150001+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:57.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:04:58.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:04:58.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:04:58.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:04:58.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:04:58.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:04:58.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:04:58.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:04:58.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T21:04:57.183462+0000 '>' 2026-02-22T20:56:13.150001+0000 2026-02-22T21:04:58.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T21:04:58.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:400: _scrub_abort: perf_counters td/osd-scrub-test 3 2026-02-22T21:04:58.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-22T21:04:58.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-22T21:04:58.534 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-22T21:04:58.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-22T21:04:58.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:04:58.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-22T21:04:58.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:04:58.618 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:04:58.619 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:04:58.619 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:04:58.619 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:04:58.619 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:04:58.619 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:04:58.619 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:04:58.619 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:04:58.619 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:04:58.620 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:04:58.620 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:04:58.620 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:04:58.620 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:04:58.620 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.620 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.620 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:04:58.620 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:04:58.620 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:04:58.620 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:04:58.621 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:04:58.621 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:04:58.621 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:04:58.621 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:04:58.621 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:04:58.621 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:04:58.621 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:04:58.621 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:04:58.621 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:04:58.621 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.621 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.622 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:04:58.622 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:04:58.622 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:04:58.622 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:04:58.622 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:04:58.622 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:04:58.622 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:04:58.622 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:04:58.622 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:04:58.622 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:04:58.622 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:04:58.623 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:04:58.623 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:04:58.623 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.623 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.623 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:04:58.623 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:04:58.623 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:04:58.623 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:04:58.623 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:04:58.623 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:04:58.624 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:04:58.624 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:04:58.624 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:04:58.624 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:04:58.624 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:04:58.624 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:04:58.624 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:04:58.624 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.624 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.624 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:04:58.624 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:04:58.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:04:58.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-22T21:04:58.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:04:58.692 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:04:58.692 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:04:58.693 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:04:58.693 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:04:58.693 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:04:58.693 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:04:58.693 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:04:58.693 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:04:58.693 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:04:58.693 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:04:58.693 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:04:58.693 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:04:58.693 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:04:58.694 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.694 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.694 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:04:58.694 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:04:58.694 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:04:58.694 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:04:58.694 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:04:58.694 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:04:58.694 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:04:58.694 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:04:58.694 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:04:58.695 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 67, 2026-02-22T21:04:58.695 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:04:58.695 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:04:58.695 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:04:58.695 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.695 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.695 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:04:58.695 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:04:58.695 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:04:58.696 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:04:58.696 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:04:58.696 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:04:58.696 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:04:58.696 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:04:58.696 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:04:58.696 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:04:58.696 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:04:58.697 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:04:58.697 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:04:58.697 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.697 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.697 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:04:58.697 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:04:58.698 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:04:58.698 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:04:58.698 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:04:58.698 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:04:58.698 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:04:58.698 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:04:58.698 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:04:58.699 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:04:58.699 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:04:58.699 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:04:58.699 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:04:58.699 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.699 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.699 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:04:58.700 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:04:58.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:04:58.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-22T21:04:58.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.762 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:04:58.763 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:04:58.764 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Teardown Test TEST_deep_scrub_abort ------------------ 2026-02-22T21:04:58.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_deep_scrub_abort ------------------' 2026-02-22T21:04:58.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-22T21:04:58.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:04:58.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:04:58.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:04:58.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:04:58.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:04:58.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:04:58.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:04:58.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:04:58.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:04:58.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:04:58.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:04:58.873 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:04:58.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:04:58.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:04:58.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:04:58.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:04:58.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:04:58.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:04:58.876 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:04:58.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:04:58.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:04:58.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:04:58.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:04:58.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:04:58.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:04:58.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:04:58.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:04:58.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:04:58.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_deep_scrub_abort ------------------' 2026-02-22T21:04:58.934 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Complete Test TEST_deep_scrub_abort ------------------ 2026-02-22T21:04:58.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-22T21:04:58.934 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Prepare Test TEST_dump_scrub_schedule ------------------- 2026-02-22T21:04:58.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_dump_scrub_schedule -------------------' 2026-02-22T21:04:58.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-22T21:04:58.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-22T21:04:58.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-22T21:04:58.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:04:58.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:04:58.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:04:58.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:04:58.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:04:58.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:04:58.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:04:58.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:04:58.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:04:58.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:04:58.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:04:58.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:04:58.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:04:58.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:04:58.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:04:58.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:04:58.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:04:58.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:04:58.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:04:58.942 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:04:58.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:04:58.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:04:58.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:04:58.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:04:58.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:04:58.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:04:58.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:04:58.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:04:58.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-22T21:04:58.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T21:04:58.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:04:58.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:04:58.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.516403 2026-02-22T21:04:58.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T21:04:58.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:04:58.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T21:04:58.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-22T21:04:58.949 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Run Test TEST_dump_scrub_schedule ----------------------- 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_dump_scrub_schedule -----------------------' 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_dump_scrub_schedule td/osd-scrub-test 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:530: TEST_dump_scrub_schedule: local dir=td/osd-scrub-test 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:531: TEST_dump_scrub_schedule: local poolname=test 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:532: TEST_dump_scrub_schedule: local OSDS=3 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:533: TEST_dump_scrub_schedule: local objects=90 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:535: TEST_dump_scrub_schedule: TESTDATA=testdata.516403 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:537: TEST_dump_scrub_schedule: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-22T21:04:58.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-22T21:04:58.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T21:04:58.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:04:58.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:04:58.972 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:04:58.972 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:04:58.972 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:04:58.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:04:58.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-22T21:04:58.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T21:04:59.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T21:04:59.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:04:59.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:04:59.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T21:04:59.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T21:04:59.002 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:04:59.002 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:04:59.002 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:04:59.002 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:04:59.002 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:04:59.002 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:04:59.002 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T21:04:59.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:04:59.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get fsid 2026-02-22T21:04:59.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T21:04:59.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:04:59.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:04:59.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T21:04:59.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T21:04:59.060 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:04:59.060 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:04:59.060 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:04:59.060 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:04:59.060 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:04:59.060 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:04:59.060 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T21:04:59.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:04:59.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get mon_host 2026-02-22T21:04:59.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:538: TEST_dump_scrub_schedule: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-22T21:04:59.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-22T21:04:59.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T21:04:59.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T21:04:59.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T21:04:59.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-22T21:04:59.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:04:59.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T21:04:59.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:04:59.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:04:59.207 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:04:59.207 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:04:59.207 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:04:59.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:04:59.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:04:59.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-22T21:04:59.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:552: TEST_dump_scrub_schedule: local 'ceph_osd_args=--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2' 2026-02-22T21:04:59.227 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: expr 3 - 1 2026-02-22T21:04:59.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: seq 0 2 2026-02-22T21:04:59.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:04:59.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:556: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 0 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-22T21:04:59.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:04:59.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:04:59.230 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:04:59.231 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:04:59.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:04:59.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:04:59.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:04:59.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:04:59.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:04:59.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:04:59.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:04:59.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:04:59.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:04:59.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:04:59.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:04:59.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:04:59.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2' 2026-02-22T21:04:59.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-22T21:04:59.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:04:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b54a0e68-b233-42d2-9861-a189edff8626 2026-02-22T21:04:59.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b54a0e68-b233-42d2-9861-a189edff8626' 2026-02-22T21:04:59.233 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 b54a0e68-b233-42d2-9861-a189edff8626 2026-02-22T21:04:59.233 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:04:59.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB7b5tpiOhLDhAAfCL9FkqBOzXRWGRW2qDAkA== 2026-02-22T21:04:59.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB7b5tpiOhLDhAAfCL9FkqBOzXRWGRW2qDAkA=="}' 2026-02-22T21:04:59.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b54a0e68-b233-42d2-9861-a189edff8626 -i td/osd-scrub-test/0/new.json 2026-02-22T21:04:59.335 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:04:59.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-22T21:04:59.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 --mkfs --key AQB7b5tpiOhLDhAAfCL9FkqBOzXRWGRW2qDAkA== --osd-uuid b54a0e68-b233-42d2-9861-a189edff8626 2026-02-22T21:04:59.363 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:04:59.352+0000 7fd9cc419a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:04:59.365 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:04:59.356+0000 7fd9cc419a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:04:59.366 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:04:59.356+0000 7fd9cc419a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:04:59.366 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:04:59.356+0000 7fd9cc419a40 -1 bdev(0x55b20dc5ec00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:04:59.366 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:04:59.356+0000 7fd9cc419a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-22T21:05:00.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-22T21:05:00.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:05:00.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:05:00.116 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T21:05:00.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:05:00.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T21:05:00.319 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T21:05:00.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-22T21:05:00.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:05:00.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:05:00.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:05:00.347 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:00.336+0000 7f2600944a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:00.349 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:00.340+0000 7f2600944a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:00.350 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:00.340+0000 7f2600944a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:00.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T21:05:00.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:05:00.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T21:05:00.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:05:00.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:05:00.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:05:00.479 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:05:00.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:05:00.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:05:00.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:05:00.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:05:00.754 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:00.744+0000 7f2600944a40 -1 Falling back to public interface 2026-02-22T21:05:01.386 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:01.376+0000 7f2600944a40 -1 osd.0 0 log_to_monitors true 2026-02-22T21:05:01.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:05:01.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:05:01.641 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:05:01.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:05:01.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:05:01.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:05:01.797 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3467002660,v1:127.0.0.1:6803/3467002660] [v2:127.0.0.1:6804/3467002660,v1:127.0.0.1:6805/3467002660] exists,up b54a0e68-b233-42d2-9861-a189edff8626 2026-02-22T21:05:01.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:05:01.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:05:01.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:05:01.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:05:01.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:556: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 1 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-22T21:05:01.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:05:01.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:05:01.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T21:05:01.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:05:01.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-22T21:05:01.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:05:01.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:05:01.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:05:01.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:05:01.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-22T21:05:01.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-22T21:05:01.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:05:01.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:05:01.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:05:01.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:05:01.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:05:01.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:05:01.803 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:05:01.803 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:01.803 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:01.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:05:01.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:05:01.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:05:01.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:05:01.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:05:01.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:05:01.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:05:01.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:05:01.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:05:01.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:05:01.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:05:01.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:05:01.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2' 2026-02-22T21:05:01.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-22T21:05:01.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:05:01.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=50d6ac56-c76c-44a6-a7cf-8226f2622f21 2026-02-22T21:05:01.805 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 50d6ac56-c76c-44a6-a7cf-8226f2622f21 2026-02-22T21:05:01.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 50d6ac56-c76c-44a6-a7cf-8226f2622f21' 2026-02-22T21:05:01.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:05:01.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB9b5tpbZw5MBAAF3PoWW/o9bSgRqLEqLwwgA== 2026-02-22T21:05:01.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB9b5tpbZw5MBAAF3PoWW/o9bSgRqLEqLwwgA=="}' 2026-02-22T21:05:01.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 50d6ac56-c76c-44a6-a7cf-8226f2622f21 -i td/osd-scrub-test/1/new.json 2026-02-22T21:05:01.975 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:05:01.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-22T21:05:01.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 --mkfs --key AQB9b5tpbZw5MBAAF3PoWW/o9bSgRqLEqLwwgA== --osd-uuid 50d6ac56-c76c-44a6-a7cf-8226f2622f21 2026-02-22T21:05:02.005 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:01.996+0000 7f66faf08a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:02.006 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:01.996+0000 7f66faf08a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:02.007 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:01.996+0000 7f66faf08a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:02.008 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:01.996+0000 7f66faf08a40 -1 bdev(0x559883388c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:05:02.008 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:02.000+0000 7f66faf08a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-22T21:05:02.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-22T21:05:02.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:05:02.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:05:02.660 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T21:05:02.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:05:02.854 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T21:05:02.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T21:05:02.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-22T21:05:02.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:05:02.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:05:02.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:05:02.882 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:02.872+0000 7f4451e8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:02.884 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:02.876+0000 7f4451e8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:02.885 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:02.876+0000 7f4451e8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:03.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T21:05:03.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:05:03.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T21:05:03.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:05:03.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:05:03.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:05:03.013 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:05:03.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:05:03.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:05:03.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:05:03.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:05:03.266 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:03.256+0000 7f4451e8fa40 -1 Falling back to public interface 2026-02-22T21:05:03.918 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:03.908+0000 7f4451e8fa40 -1 osd.1 0 log_to_monitors true 2026-02-22T21:05:04.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:05:04.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:05:04.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:05:04.179 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:05:04.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:05:04.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:05:04.334 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2820708162,v1:127.0.0.1:6811/2820708162] [v2:127.0.0.1:6812/2820708162,v1:127.0.0.1:6813/2820708162] exists,up 50d6ac56-c76c-44a6-a7cf-8226f2622f21 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:554: TEST_dump_scrub_schedule: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:556: TEST_dump_scrub_schedule: run_osd td/osd-scrub-test 2 --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:05:04.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:05:04.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:05:04.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:05:04.336 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:05:04.336 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:04.336 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:04.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:05:04.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:05:04.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:05:04.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:05:04.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:05:04.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:05:04.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:05:04.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:05:04.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:05:04.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:05:04.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:05:04.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:05:04.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2' 2026-02-22T21:05:04.337 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-22T21:05:04.338 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:05:04.339 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e236a127-a0af-40e6-a9fb-5b1440523786 2026-02-22T21:05:04.339 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 e236a127-a0af-40e6-a9fb-5b1440523786 2026-02-22T21:05:04.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 e236a127-a0af-40e6-a9fb-5b1440523786' 2026-02-22T21:05:04.340 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:05:04.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCAb5tpiHCjFBAAqUNjlizP6I/wAxUHVgfyeg== 2026-02-22T21:05:04.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCAb5tpiHCjFBAAqUNjlizP6I/wAxUHVgfyeg=="}' 2026-02-22T21:05:04.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e236a127-a0af-40e6-a9fb-5b1440523786 -i td/osd-scrub-test/2/new.json 2026-02-22T21:05:04.510 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:05:04.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-22T21:05:04.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 --mkfs --key AQCAb5tpiHCjFBAAqUNjlizP6I/wAxUHVgfyeg== --osd-uuid e236a127-a0af-40e6-a9fb-5b1440523786 2026-02-22T21:05:04.539 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:04.528+0000 7f732cd28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:04.541 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:04.532+0000 7f732cd28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:04.542 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:04.532+0000 7f732cd28a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:04.542 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:04.532+0000 7f732cd28a40 -1 bdev(0x560adcf38c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:05:04.543 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:04.532+0000 7f732cd28a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-22T21:05:05.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-22T21:05:05.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:05:05.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:05:05.160 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T21:05:05.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:05:05.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T21:05:05.367 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T21:05:05.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_deep_scrub_randomize_ratio=0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_backoff_ratio=0.0 --osd_op_queue=wpq --osd_stats_update_period_not_scrubbing=1 --osd_stats_update_period_scrubbing=1 --osd_scrub_retry_after_noscrub=1 --osd_scrub_retry_pg_state=2 --osd_scrub_retry_delay=2 --osd_scrub_sleep=0.2 2026-02-22T21:05:05.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:05:05.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:05:05.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:05:05.385 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:05.376+0000 7f603d9c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:05.386 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:05.376+0000 7f603d9c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:05.387 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:05.376+0000 7f603d9c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:05.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T21:05:05.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:05:05.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T21:05:05.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:05:05.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:05:05.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:05:05.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:05:05.525 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:05:05.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:05:05.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:05:05.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:05:05.793 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:05.784+0000 7f603d9c5a40 -1 Falling back to public interface 2026-02-22T21:05:06.371 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:06.360+0000 7f603d9c5a40 -1 osd.2 0 log_to_monitors true 2026-02-22T21:05:06.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:05:06.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:05:06.682 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:05:06.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:05:06.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:05:06.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:05:06.841 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2068068992,v1:127.0.0.1:6819/2068068992] [v2:127.0.0.1:6820/2068068992,v1:127.0.0.1:6821/2068068992] exists,up e236a127-a0af-40e6-a9fb-5b1440523786 2026-02-22T21:05:06.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:05:06.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:05:06.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:05:06.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:560: TEST_dump_scrub_schedule: create_pool test 1 1 2026-02-22T21:05:06.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-22T21:05:07.084 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T21:05:07.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T21:05:08.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:561: TEST_dump_scrub_schedule: wait_for_clean 2026-02-22T21:05:08.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:05:08.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:05:08.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:05:08.096 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:05:08.097 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:05:08.097 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:05:08.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:05:08.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:05:08.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:05:08.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:05:08.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:05:08.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:05:08.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:05:08.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:05:08.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:05:08.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:05:08.363 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:05:08.363 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:05:08.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:05:08.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:05:08.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:05:08.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-22T21:05:08.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-22T21:05:08.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-22T21:05:08.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:05:08.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:05:08.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-22T21:05:08.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-22T21:05:08.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-38654705669' 2026-02-22T21:05:08.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:05:08.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:05:08.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-22T21:05:08.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-22T21:05:08.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-38654705669 2-60129542147' 2026-02-22T21:05:08.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:05:08.582 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-22T21:05:08.582 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:05:08.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:05:08.584 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-22T21:05:08.584 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:05:08.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-22T21:05:08.586 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836487 2026-02-22T21:05:08.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-22T21:05:08.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:05:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T21:05:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:05:09.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:05:09.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:05:09.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836487 2026-02-22T21:05:09.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:05:09.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-22T21:05:09.915 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:05:09.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:05:09.917 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-22T21:05:09.917 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:05:09.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-22T21:05:09.918 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705669 2026-02-22T21:05:09.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-22T21:05:09.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:05:10.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705670 -lt 38654705669 2026-02-22T21:05:10.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:05:10.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-22T21:05:10.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:05:10.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:05:10.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-22T21:05:10.083 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:05:10.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-22T21:05:10.084 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542147 2026-02-22T21:05:10.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-22T21:05:10.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:05:10.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542147 2026-02-22T21:05:10.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:05:10.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:05:10.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:05:10.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:05:10.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:05:10.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:05:10.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:05:10.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:05:10.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:05:10.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:05:10.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:05:10.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:05:10.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:05:10.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:05:10.588 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:05:10.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:05:10.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:05:10.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:05:10.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: ceph osd dump 2026-02-22T21:05:10.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: grep '^pool.*['\'']test['\'']' 2026-02-22T21:05:10.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: awk '{ print $2 }' 2026-02-22T21:05:10.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:562: TEST_dump_scrub_schedule: poolid=1 2026-02-22T21:05:10.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:564: TEST_dump_scrub_schedule: dd if=/dev/urandom of=testdata.516403 bs=1032 count=1 2026-02-22T21:05:10.933 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T21:05:10.933 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T21:05:10.933 INFO:tasks.workunit.client.0.trial150.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.5577e-05 s, 29.0 MB/s 2026-02-22T21:05:10.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: seq 1 90 2026-02-22T21:05:10.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:10.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj1 testdata.516403 2026-02-22T21:05:10.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:10.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj2 testdata.516403 2026-02-22T21:05:10.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:10.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj3 testdata.516403 2026-02-22T21:05:11.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj4 testdata.516403 2026-02-22T21:05:11.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj5 testdata.516403 2026-02-22T21:05:11.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj6 testdata.516403 2026-02-22T21:05:11.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj7 testdata.516403 2026-02-22T21:05:11.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj8 testdata.516403 2026-02-22T21:05:11.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj9 testdata.516403 2026-02-22T21:05:11.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj10 testdata.516403 2026-02-22T21:05:11.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj11 testdata.516403 2026-02-22T21:05:11.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj12 testdata.516403 2026-02-22T21:05:11.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj13 testdata.516403 2026-02-22T21:05:11.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj14 testdata.516403 2026-02-22T21:05:11.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj15 testdata.516403 2026-02-22T21:05:11.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj16 testdata.516403 2026-02-22T21:05:11.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj17 testdata.516403 2026-02-22T21:05:11.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj18 testdata.516403 2026-02-22T21:05:11.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj19 testdata.516403 2026-02-22T21:05:11.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj20 testdata.516403 2026-02-22T21:05:11.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj21 testdata.516403 2026-02-22T21:05:11.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj22 testdata.516403 2026-02-22T21:05:11.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj23 testdata.516403 2026-02-22T21:05:11.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj24 testdata.516403 2026-02-22T21:05:11.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj25 testdata.516403 2026-02-22T21:05:11.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj26 testdata.516403 2026-02-22T21:05:11.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj27 testdata.516403 2026-02-22T21:05:11.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj28 testdata.516403 2026-02-22T21:05:11.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.603 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj29 testdata.516403 2026-02-22T21:05:11.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj30 testdata.516403 2026-02-22T21:05:11.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj31 testdata.516403 2026-02-22T21:05:11.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj32 testdata.516403 2026-02-22T21:05:11.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj33 testdata.516403 2026-02-22T21:05:11.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj34 testdata.516403 2026-02-22T21:05:11.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj35 testdata.516403 2026-02-22T21:05:11.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj36 testdata.516403 2026-02-22T21:05:11.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj37 testdata.516403 2026-02-22T21:05:11.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj38 testdata.516403 2026-02-22T21:05:11.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj39 testdata.516403 2026-02-22T21:05:11.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj40 testdata.516403 2026-02-22T21:05:11.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj41 testdata.516403 2026-02-22T21:05:11.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj42 testdata.516403 2026-02-22T21:05:11.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj43 testdata.516403 2026-02-22T21:05:11.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj44 testdata.516403 2026-02-22T21:05:11.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:11.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj45 testdata.516403 2026-02-22T21:05:12.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj46 testdata.516403 2026-02-22T21:05:12.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj47 testdata.516403 2026-02-22T21:05:12.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj48 testdata.516403 2026-02-22T21:05:12.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj49 testdata.516403 2026-02-22T21:05:12.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj50 testdata.516403 2026-02-22T21:05:12.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj51 testdata.516403 2026-02-22T21:05:12.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj52 testdata.516403 2026-02-22T21:05:12.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj53 testdata.516403 2026-02-22T21:05:12.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj54 testdata.516403 2026-02-22T21:05:12.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj55 testdata.516403 2026-02-22T21:05:12.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj56 testdata.516403 2026-02-22T21:05:12.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj57 testdata.516403 2026-02-22T21:05:12.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj58 testdata.516403 2026-02-22T21:05:12.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj59 testdata.516403 2026-02-22T21:05:12.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj60 testdata.516403 2026-02-22T21:05:12.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj61 testdata.516403 2026-02-22T21:05:12.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj62 testdata.516403 2026-02-22T21:05:12.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj63 testdata.516403 2026-02-22T21:05:12.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj64 testdata.516403 2026-02-22T21:05:12.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj65 testdata.516403 2026-02-22T21:05:12.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj66 testdata.516403 2026-02-22T21:05:12.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj67 testdata.516403 2026-02-22T21:05:12.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj68 testdata.516403 2026-02-22T21:05:12.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj69 testdata.516403 2026-02-22T21:05:12.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj70 testdata.516403 2026-02-22T21:05:12.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj71 testdata.516403 2026-02-22T21:05:12.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj72 testdata.516403 2026-02-22T21:05:12.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj73 testdata.516403 2026-02-22T21:05:12.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj74 testdata.516403 2026-02-22T21:05:12.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj75 testdata.516403 2026-02-22T21:05:12.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj76 testdata.516403 2026-02-22T21:05:12.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj77 testdata.516403 2026-02-22T21:05:12.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj78 testdata.516403 2026-02-22T21:05:12.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj79 testdata.516403 2026-02-22T21:05:12.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj80 testdata.516403 2026-02-22T21:05:12.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj81 testdata.516403 2026-02-22T21:05:12.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj82 testdata.516403 2026-02-22T21:05:12.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj83 testdata.516403 2026-02-22T21:05:12.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj84 testdata.516403 2026-02-22T21:05:12.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj85 testdata.516403 2026-02-22T21:05:12.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:12.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj86 testdata.516403 2026-02-22T21:05:13.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:13.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj87 testdata.516403 2026-02-22T21:05:13.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:13.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj88 testdata.516403 2026-02-22T21:05:13.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:13.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj89 testdata.516403 2026-02-22T21:05:13.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:565: TEST_dump_scrub_schedule: for i in `seq 1 $objects` 2026-02-22T21:05:13.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:567: TEST_dump_scrub_schedule: rados -p test put obj90 testdata.516403 2026-02-22T21:05:13.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:569: TEST_dump_scrub_schedule: rm -f testdata.516403 2026-02-22T21:05:13.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:571: TEST_dump_scrub_schedule: local pgid=1.0 2026-02-22T21:05:13.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:573: TEST_dump_scrub_schedule: date +%Y-%m-%dT%H:%M:%S.%N%:z 2026-02-22T21:05:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:573: TEST_dump_scrub_schedule: local now_is=2026-02-22T21:05:13.100548947+00:00 2026-02-22T21:05:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:581: TEST_dump_scrub_schedule: expct_starting=(['query_active']='false' ['query_is_future']='true' ['query_schedule']='scrub scheduled') 2026-02-22T21:05:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:581: TEST_dump_scrub_schedule: declare -A expct_starting 2026-02-22T21:05:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:582: TEST_dump_scrub_schedule: declare -A sched_data 2026-02-22T21:05:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:583: TEST_dump_scrub_schedule: extract_published_sch 1.0 2026-02-22T21:05:13.100548947+00:00 2019-10-12T20:32:43.645168+0000 sched_data 2026-02-22T21:05:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.0 2026-02-22T21:05:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2026-02-22T21:05:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2026-02-22T21:05:13.100548947+00:00 2026-02-22T21:05:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2019-10-12T20:32:43.645168+0000 2026-02-22T21:05:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2026-02-22T21:05:13.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2026-02-22T21:05:13.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2026-02-22T21:05:13.250 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:05:13.260 INFO:tasks.workunit.client.0.trial150.stdout:from pg dump pg: { 2026-02-22T21:05:13.261 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_pg_state": "active+clean", 2026-02-22T21:05:13.261 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_state_has_scrubbing": false, 2026-02-22T21:05:13.261 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_last_duration": 0, 2026-02-22T21:05:13.261 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-22T21:05:13.261 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule_at": "2026-02-23T21:05:07.077569+0000", 2026-02-22T21:05:13.261 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_is_future": true, 2026-02-22T21:05:13.261 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_vs_date": true, 2026-02-22T21:05:13.261 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_reported_epoch": 18, 2026-02-22T21:05:13.261 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_seq": 22 2026-02-22T21:05:13.261 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:13.262 INFO:tasks.workunit.client.0.trial150.stdout:query output: 2026-02-22T21:05:13.332 INFO:tasks.workunit.client.0.trial150.stdout: "scrubber": { 2026-02-22T21:05:13.332 INFO:tasks.workunit.client.0.trial150.stdout: "active": false, 2026-02-22T21:05:13.332 INFO:tasks.workunit.client.0.trial150.stdout: "must_scrub": false, 2026-02-22T21:05:13.333 INFO:tasks.workunit.client.0.trial150.stdout: "must_deep_scrub": false, 2026-02-22T21:05:13.333 INFO:tasks.workunit.client.0.trial150.stdout: "must_repair": false, 2026-02-22T21:05:13.333 INFO:tasks.workunit.client.0.trial150.stdout: "scrub_reg_stamp": "2026-02-23T21:05:07.077569+0000", 2026-02-22T21:05:13.333 INFO:tasks.workunit.client.0.trial150.stdout: "schedule": "scrub scheduled @ 2026-02-23T21:05:07.077 (2026-02-23T21:05:07.077)" 2026-02-22T21:05:13.333 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:13.333 INFO:tasks.workunit.client.0.trial150.stdout: "agent_state": {} 2026-02-22T21:05:13.418 INFO:tasks.workunit.client.0.trial150.stdout:( 2026-02-22T21:05:13.418 INFO:tasks.workunit.client.0.trial150.stdout:[query_epoch]=18 2026-02-22T21:05:13.418 INFO:tasks.workunit.client.0.trial150.stdout:[query_seq]=108 2026-02-22T21:05:13.418 INFO:tasks.workunit.client.0.trial150.stdout:[query_active]=false 2026-02-22T21:05:13.418 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule]='scrub scheduled' 2026-02-22T21:05:13.418 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule_at]='2026-02-23T21:05:07.077' 2026-02-22T21:05:13.418 INFO:tasks.workunit.client.0.trial150.stdout:[query_target_at]='2026-02-23T21:05:07.077' 2026-02-22T21:05:13.418 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_duration]=0 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_stamp]='2026-02-22T21:05:07.077569+0000' 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_scrub]='0x0' 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[query_is_future]=true 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[query_vs_date]=true 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[query_scrub_seq]=null 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_pg_state]='active+clean' 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_state_has_scrubbing]=false 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_last_duration]=0 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule_at]='2026-02-23T21:05:07.077569+0000' 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_is_future]=true 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_vs_date]=true 2026-02-22T21:05:13.419 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_reported_epoch]=18 2026-02-22T21:05:13.420 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_seq]=22 2026-02-22T21:05:13.420 INFO:tasks.workunit.client.0.trial150.stdout:) 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:584: TEST_dump_scrub_schedule: schedule_against_expected sched_data expct_starting initial 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:178: schedule_against_expected: local -n dict=sched_data 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:179: schedule_against_expected: local -n ep=expct_starting 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:180: schedule_against_expected: local extr_dbg=1 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:183: schedule_against_expected: local saved_echo_flag=x 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:184: schedule_against_expected: set +x 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stdout:-- - comparing: 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stdout:key is query_schedule expected: scrub scheduled in actual: scrub scheduled 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stdout:key is query_is_future expected: true in actual: true 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stdout:key is query_active expected: false in actual: false 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:203: schedule_against_expected: return 0 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:585: TEST_dump_scrub_schedule: (( 0 == 0 )) 2026-02-22T21:05:13.429 INFO:tasks.workunit.client.0.trial150.stdout:last-scrub --- 0x0 2026-02-22T21:05:13.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:586: TEST_dump_scrub_schedule: echo 'last-scrub --- ' 0x0 2026-02-22T21:05:13.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:592: TEST_dump_scrub_schedule: saved_last_stamp=2026-02-22T21:05:07.077569+0000 2026-02-22T21:05:13.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:593: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 0 2026-02-22T21:05:13.493 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T21:05:13.493 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T21:05:13.493 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:13.500 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T21:05:13.500 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T21:05:13.500 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:13.507 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T21:05:13.507 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T21:05:13.507 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:13.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:594: TEST_dump_scrub_schedule: ceph tell 1.0 deep-scrub 2026-02-22T21:05:13.579 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:05:13.579 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T21:05:13.579 INFO:tasks.workunit.client.0.trial150.stdout: "must": true, 2026-02-22T21:05:13.579 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "0.000000" 2026-02-22T21:05:13.579 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:13.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:599: TEST_dump_scrub_schedule: sleep 5 2026-02-22T21:05:18.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:600: TEST_dump_scrub_schedule: sched_data=() 2026-02-22T21:05:18.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:601: TEST_dump_scrub_schedule: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2026-02-22T21:05:18.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:601: TEST_dump_scrub_schedule: declare -A expct_qry_duration 2026-02-22T21:05:18.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:602: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-22T21:05:07.077569+0000 expct_qry_duration 'WaitingAfterScrub ' sched_data 2026-02-22T21:05:18.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-22T21:05:18.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-22T21:05:18.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-22T21:05:07.077569+0000 2026-02-22T21:05:18.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_qry_duration 2026-02-22T21:05:18.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-22T21:05:18.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-22T21:05:18.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-22T21:05:18.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-22T21:05:18.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-22T21:05:18.590 INFO:tasks.workunit.client.0.trial150.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.0 dt:2026-02-22T21:05:07.077569+0000 (10 retries) 2026-02-22T21:05:19.239 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:05:19.249 INFO:tasks.workunit.client.0.trial150.stdout:from pg dump pg: { 2026-02-22T21:05:19.249 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_pg_state": "active+clean", 2026-02-22T21:05:19.249 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_state_has_scrubbing": false, 2026-02-22T21:05:19.249 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_last_duration": 1, 2026-02-22T21:05:19.249 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-22T21:05:19.249 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule_at": "2026-02-23T21:05:13.934587+0000", 2026-02-22T21:05:19.249 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_is_future": true, 2026-02-22T21:05:19.249 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_vs_date": true, 2026-02-22T21:05:19.249 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_reported_epoch": 18, 2026-02-22T21:05:19.249 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_seq": 124 2026-02-22T21:05:19.249 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:19.249 INFO:tasks.workunit.client.0.trial150.stdout:query output: 2026-02-22T21:05:19.320 INFO:tasks.workunit.client.0.trial150.stdout: "scrubber": { 2026-02-22T21:05:19.320 INFO:tasks.workunit.client.0.trial150.stdout: "active": false, 2026-02-22T21:05:19.320 INFO:tasks.workunit.client.0.trial150.stdout: "must_scrub": false, 2026-02-22T21:05:19.320 INFO:tasks.workunit.client.0.trial150.stdout: "must_deep_scrub": false, 2026-02-22T21:05:19.320 INFO:tasks.workunit.client.0.trial150.stdout: "must_repair": false, 2026-02-22T21:05:19.320 INFO:tasks.workunit.client.0.trial150.stdout: "scrub_reg_stamp": "2026-02-23T21:05:13.934587+0000", 2026-02-22T21:05:19.320 INFO:tasks.workunit.client.0.trial150.stdout: "schedule": "scrub scheduled @ 2026-02-23T21:05:13.934 (2026-02-23T21:05:13.934)" 2026-02-22T21:05:19.320 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:19.320 INFO:tasks.workunit.client.0.trial150.stdout: "agent_state": {} 2026-02-22T21:05:19.402 INFO:tasks.workunit.client.0.trial150.stdout:( 2026-02-22T21:05:19.402 INFO:tasks.workunit.client.0.trial150.stdout:[query_epoch]=18 2026-02-22T21:05:19.402 INFO:tasks.workunit.client.0.trial150.stdout:[query_seq]=126 2026-02-22T21:05:19.402 INFO:tasks.workunit.client.0.trial150.stdout:[query_active]=false 2026-02-22T21:05:19.402 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule]='scrub scheduled' 2026-02-22T21:05:19.402 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule_at]='2026-02-23T21:05:13.934' 2026-02-22T21:05:19.402 INFO:tasks.workunit.client.0.trial150.stdout:[query_target_at]='2026-02-23T21:05:13.934' 2026-02-22T21:05:19.402 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_duration]=1 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_stamp]='2026-02-22T21:05:13.934587+0000' 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_scrub]='18x90' 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[query_is_future]=true 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[query_vs_date]=true 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[query_scrub_seq]=null 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_pg_state]='active+clean' 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_state_has_scrubbing]=false 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_last_duration]=1 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule_at]='2026-02-23T21:05:13.934587+0000' 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_is_future]=true 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_vs_date]=true 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_reported_epoch]=18 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_seq]=124 2026-02-22T21:05:19.403 INFO:tasks.workunit.client.0.trial150.stdout:) 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stdout:--> loop: 1 ~ false / 126 / 124 / true / 2026-02-22T21:05:13.934587+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:604: TEST_dump_scrub_schedule: sched_data=() 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:605: TEST_dump_scrub_schedule: expct_dmp_duration=(['dmp_last_duration']='0' ['dmp_last_duration_neg']='not0') 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:605: TEST_dump_scrub_schedule: declare -A expct_dmp_duration 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:606: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-22T21:05:07.077569+0000 expct_dmp_duration 'WaitingAfterScrub_dmp ' sched_data 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-22T21:05:07.077569+0000 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_dmp_duration 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-22T21:05:19.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-22T21:05:19.414 INFO:tasks.workunit.client.0.trial150.stdout:waiting for any condition (WaitingAfterScrub_dmp ): pg:1.0 dt:2026-02-22T21:05:07.077569+0000 (10 retries) 2026-02-22T21:05:20.063 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:05:20.074 INFO:tasks.workunit.client.0.trial150.stdout:from pg dump pg: { 2026-02-22T21:05:20.074 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_pg_state": "active+clean", 2026-02-22T21:05:20.074 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_state_has_scrubbing": false, 2026-02-22T21:05:20.074 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_last_duration": 1, 2026-02-22T21:05:20.074 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-22T21:05:20.074 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule_at": "2026-02-23T21:05:13.934587+0000", 2026-02-22T21:05:20.074 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_is_future": true, 2026-02-22T21:05:20.074 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_vs_date": true, 2026-02-22T21:05:20.074 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_reported_epoch": 18, 2026-02-22T21:05:20.074 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_seq": 125 2026-02-22T21:05:20.074 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:20.074 INFO:tasks.workunit.client.0.trial150.stdout:query output: 2026-02-22T21:05:20.148 INFO:tasks.workunit.client.0.trial150.stdout: "scrubber": { 2026-02-22T21:05:20.148 INFO:tasks.workunit.client.0.trial150.stdout: "active": false, 2026-02-22T21:05:20.148 INFO:tasks.workunit.client.0.trial150.stdout: "must_scrub": false, 2026-02-22T21:05:20.148 INFO:tasks.workunit.client.0.trial150.stdout: "must_deep_scrub": false, 2026-02-22T21:05:20.148 INFO:tasks.workunit.client.0.trial150.stdout: "must_repair": false, 2026-02-22T21:05:20.148 INFO:tasks.workunit.client.0.trial150.stdout: "scrub_reg_stamp": "2026-02-23T21:05:13.934587+0000", 2026-02-22T21:05:20.148 INFO:tasks.workunit.client.0.trial150.stdout: "schedule": "scrub scheduled @ 2026-02-23T21:05:13.934 (2026-02-23T21:05:13.934)" 2026-02-22T21:05:20.148 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:20.148 INFO:tasks.workunit.client.0.trial150.stdout: "agent_state": {} 2026-02-22T21:05:20.232 INFO:tasks.workunit.client.0.trial150.stdout:( 2026-02-22T21:05:20.232 INFO:tasks.workunit.client.0.trial150.stdout:[query_epoch]=18 2026-02-22T21:05:20.232 INFO:tasks.workunit.client.0.trial150.stdout:[query_seq]=126 2026-02-22T21:05:20.232 INFO:tasks.workunit.client.0.trial150.stdout:[query_active]=false 2026-02-22T21:05:20.232 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule]='scrub scheduled' 2026-02-22T21:05:20.232 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule_at]='2026-02-23T21:05:13.934' 2026-02-22T21:05:20.232 INFO:tasks.workunit.client.0.trial150.stdout:[query_target_at]='2026-02-23T21:05:13.934' 2026-02-22T21:05:20.232 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_duration]=1 2026-02-22T21:05:20.233 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_stamp]='2026-02-22T21:05:13.934587+0000' 2026-02-22T21:05:20.233 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_scrub]='18x90' 2026-02-22T21:05:20.233 INFO:tasks.workunit.client.0.trial150.stdout:[query_is_future]=true 2026-02-22T21:05:20.233 INFO:tasks.workunit.client.0.trial150.stdout:[query_vs_date]=true 2026-02-22T21:05:20.233 INFO:tasks.workunit.client.0.trial150.stdout:[query_scrub_seq]=null 2026-02-22T21:05:20.233 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_pg_state]='active+clean' 2026-02-22T21:05:20.233 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_state_has_scrubbing]=false 2026-02-22T21:05:20.233 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_last_duration]=1 2026-02-22T21:05:20.233 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-22T21:05:20.233 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule_at]='2026-02-23T21:05:13.934587+0000' 2026-02-22T21:05:20.233 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_is_future]=true 2026-02-22T21:05:20.234 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_vs_date]=true 2026-02-22T21:05:20.234 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_reported_epoch]=18 2026-02-22T21:05:20.234 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_seq]=125 2026-02-22T21:05:20.234 INFO:tasks.workunit.client.0.trial150.stdout:) 2026-02-22T21:05:20.244 INFO:tasks.workunit.client.0.trial150.stdout:--> loop: 1 ~ false / 126 / 125 / true / 2026-02-22T21:05:13.934587+0000 / scrub scheduled %%% dmp_last_duration dmp_last_duration_neg 2026-02-22T21:05:20.244 INFO:tasks.workunit.client.0.trial150.stdout:key is dmp_last_duration: negation:1 # expected: 0 # in actual: 1 2026-02-22T21:05:20.244 INFO:tasks.workunit.client.0.trial150.stdout:WaitingAfterScrub_dmp - 'dmp_last_duration' actual value (1) matches expected (0) (negation: 1) 2026-02-22T21:05:20.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-22T21:05:20.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:613: TEST_dump_scrub_schedule: ceph osd set noscrub 2026-02-22T21:05:20.443 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T21:05:20.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:614: TEST_dump_scrub_schedule: sleep 2 2026-02-22T21:05:22.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:615: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_shallow_scrub_chunk_max 3 2026-02-22T21:05:22.524 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T21:05:22.524 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T21:05:22.524 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:22.531 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T21:05:22.531 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T21:05:22.531 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:22.538 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T21:05:22.539 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_shallow_scrub_chunk_max = '' " 2026-02-22T21:05:22.539 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:22.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:616: TEST_dump_scrub_schedule: ceph tell 'osd.*' config set osd_scrub_sleep 2.0 2026-02-22T21:05:22.621 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T21:05:22.621 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T21:05:22.621 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:22.628 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T21:05:22.629 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T21:05:22.629 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:22.636 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T21:05:22.636 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_sleep = '' " 2026-02-22T21:05:22.636 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:22.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:617: TEST_dump_scrub_schedule: sleep 8 2026-02-22T21:05:30.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:618: TEST_dump_scrub_schedule: saved_last_stamp=2026-02-22T21:05:13.934587+0000 2026-02-22T21:05:30.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:620: TEST_dump_scrub_schedule: ceph tell 1.0 schedule-scrub 2026-02-22T21:05:30.709 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:05:30.710 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T21:05:30.710 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T21:05:30.710 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-20T21:05:30.705195+0000" 2026-02-22T21:05:30.710 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:30.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:621: TEST_dump_scrub_schedule: sleep 1 2026-02-22T21:05:31.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:622: TEST_dump_scrub_schedule: sched_data=() 2026-02-22T21:05:31.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:623: TEST_dump_scrub_schedule: expct_scrub_peri_sched=(['query_is_future']='false') 2026-02-22T21:05:31.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:623: TEST_dump_scrub_schedule: declare -A expct_scrub_peri_sched 2026-02-22T21:05:31.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:624: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-22T21:05:13.934587+0000 expct_scrub_peri_sched waitingBeingScheduled sched_data 2026-02-22T21:05:31.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-22T21:05:31.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-22T21:05:31.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-22T21:05:13.934587+0000 2026-02-22T21:05:31.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_scrub_peri_sched 2026-02-22T21:05:31.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-22T21:05:31.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-22T21:05:31.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-22T21:05:31.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-22T21:05:31.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-22T21:05:31.721 INFO:tasks.workunit.client.0.trial150.stdout:waiting for any condition (waitingBeingScheduled): pg:1.0 dt:2026-02-22T21:05:13.934587+0000 (10 retries) 2026-02-22T21:05:32.370 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:05:32.380 INFO:tasks.workunit.client.0.trial150.stdout:from pg dump pg: { 2026-02-22T21:05:32.380 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_pg_state": "active+clean", 2026-02-22T21:05:32.381 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_state_has_scrubbing": false, 2026-02-22T21:05:32.381 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_last_duration": 1, 2026-02-22T21:05:32.381 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-22T21:05:32.381 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule_at": "2026-02-22T21:05:31.978132+0000", 2026-02-22T21:05:32.381 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_is_future": true, 2026-02-22T21:05:32.381 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_vs_date": true, 2026-02-22T21:05:32.381 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_reported_epoch": 19, 2026-02-22T21:05:32.381 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_seq": 135 2026-02-22T21:05:32.381 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:32.381 INFO:tasks.workunit.client.0.trial150.stdout:query output: 2026-02-22T21:05:32.453 INFO:tasks.workunit.client.0.trial150.stdout: "scrubber": { 2026-02-22T21:05:32.453 INFO:tasks.workunit.client.0.trial150.stdout: "active": false, 2026-02-22T21:05:32.453 INFO:tasks.workunit.client.0.trial150.stdout: "must_scrub": false, 2026-02-22T21:05:32.453 INFO:tasks.workunit.client.0.trial150.stdout: "must_deep_scrub": false, 2026-02-22T21:05:32.453 INFO:tasks.workunit.client.0.trial150.stdout: "must_repair": false, 2026-02-22T21:05:32.453 INFO:tasks.workunit.client.0.trial150.stdout: "scrub_reg_stamp": "2026-02-22T21:05:31.978132+0000", 2026-02-22T21:05:32.453 INFO:tasks.workunit.client.0.trial150.stdout: "schedule": "queued for scrub at 2026-02-21T21:05:30.705" 2026-02-22T21:05:32.453 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:32.453 INFO:tasks.workunit.client.0.trial150.stdout: "agent_state": {} 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:( 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[query_epoch]=19 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[query_seq]=136 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[query_active]=false 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule]='queued for scrub at 2026-02-21T21:05:30.705' 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule_at]='0' 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[query_target_at]='0' 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_duration]=1 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_stamp]='2026-02-20T21:05:30.705195+0000' 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_scrub]='18x90' 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[query_is_future]=false 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[query_vs_date]=false 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[query_scrub_seq]=null 2026-02-22T21:05:32.538 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_pg_state]='active+clean' 2026-02-22T21:05:32.539 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_state_has_scrubbing]=false 2026-02-22T21:05:32.539 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_last_duration]=1 2026-02-22T21:05:32.539 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-22T21:05:32.539 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule_at]='2026-02-22T21:05:31.978132+0000' 2026-02-22T21:05:32.539 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_is_future]=true 2026-02-22T21:05:32.539 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_vs_date]=true 2026-02-22T21:05:32.539 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_reported_epoch]=19 2026-02-22T21:05:32.539 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_seq]=135 2026-02-22T21:05:32.539 INFO:tasks.workunit.client.0.trial150.stdout:) 2026-02-22T21:05:32.549 INFO:tasks.workunit.client.0.trial150.stdout:--> loop: 1 ~ false / 136 / 135 / false / 2026-02-20T21:05:30.705195+0000 / queued for scrub at 2026-02-21T21:05:30.705 %%% query_is_future 2026-02-22T21:05:32.549 INFO:tasks.workunit.client.0.trial150.stdout:key is query_is_future: negation:0 # expected: false # in actual: false 2026-02-22T21:05:32.549 INFO:tasks.workunit.client.0.trial150.stdout:waitingBeingScheduled - 'query_is_future' actual value (false) matches expected (false) (negation: 0) 2026-02-22T21:05:32.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-22T21:05:32.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:636: TEST_dump_scrub_schedule: saved_last_stamp=2026-02-20T21:05:30.705195+0000 2026-02-22T21:05:32.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:637: TEST_dump_scrub_schedule: ceph osd unset noscrub 2026-02-22T21:05:32.749 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is unset 2026-02-22T21:05:32.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:639: TEST_dump_scrub_schedule: cond_active=(['query_active']='true') 2026-02-22T21:05:32.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:639: TEST_dump_scrub_schedule: declare -A cond_active 2026-02-22T21:05:32.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:640: TEST_dump_scrub_schedule: sched_data=() 2026-02-22T21:05:32.761 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:641: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-20T21:05:30.705195+0000 cond_active 'WaitingActive ' sched_data 2026-02-22T21:05:32.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-22T21:05:32.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-22T21:05:32.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-20T21:05:30.705195+0000 2026-02-22T21:05:32.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=cond_active 2026-02-22T21:05:32.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-22T21:05:32.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-22T21:05:32.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-22T21:05:32.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-22T21:05:32.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-22T21:05:32.763 INFO:tasks.workunit.client.0.trial150.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2026-02-20T21:05:30.705195+0000 (10 retries) 2026-02-22T21:05:33.414 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:05:33.425 INFO:tasks.workunit.client.0.trial150.stdout:from pg dump pg: { 2026-02-22T21:05:33.425 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_pg_state": "active+clean", 2026-02-22T21:05:33.425 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_state_has_scrubbing": false, 2026-02-22T21:05:33.425 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_last_duration": 1, 2026-02-22T21:05:33.425 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-22T21:05:33.426 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule_at": "2026-02-22T21:05:31.978132+0000", 2026-02-22T21:05:33.426 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_is_future": false, 2026-02-22T21:05:33.426 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_vs_date": true, 2026-02-22T21:05:33.426 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_reported_epoch": 19, 2026-02-22T21:05:33.426 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_seq": 135 2026-02-22T21:05:33.426 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:33.426 INFO:tasks.workunit.client.0.trial150.stdout:query output: 2026-02-22T21:05:33.500 INFO:tasks.workunit.client.0.trial150.stdout: "scrubber": { 2026-02-22T21:05:33.500 INFO:tasks.workunit.client.0.trial150.stdout: "active": true, 2026-02-22T21:05:33.500 INFO:tasks.workunit.client.0.trial150.stdout: "epoch_start": "15", 2026-02-22T21:05:33.500 INFO:tasks.workunit.client.0.trial150.stdout: "start": "1:00000000::::0", 2026-02-22T21:05:33.500 INFO:tasks.workunit.client.0.trial150.stdout: "end": "MIN", 2026-02-22T21:05:33.500 INFO:tasks.workunit.client.0.trial150.stdout: "max_end": "MIN", 2026-02-22T21:05:33.500 INFO:tasks.workunit.client.0.trial150.stdout: "subset_last_update": "0'0", 2026-02-22T21:05:33.500 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T21:05:33.500 INFO:tasks.workunit.client.0.trial150.stdout: "req_scrub": false, 2026-02-22T21:05:33.501 INFO:tasks.workunit.client.0.trial150.stdout: "auto_repair": false, 2026-02-22T21:05:33.501 INFO:tasks.workunit.client.0.trial150.stdout: "check_repair": false, 2026-02-22T21:05:33.501 INFO:tasks.workunit.client.0.trial150.stdout: "deep_scrub_on_error": false, 2026-02-22T21:05:33.501 INFO:tasks.workunit.client.0.trial150.stdout: "priority": 5, 2026-02-22T21:05:33.501 INFO:tasks.workunit.client.0.trial150.stdout: "shallow_errors": 0, 2026-02-22T21:05:33.501 INFO:tasks.workunit.client.0.trial150.stdout: "deep_errors": 0, 2026-02-22T21:05:33.501 INFO:tasks.workunit.client.0.trial150.stdout: "fixed": 0, 2026-02-22T21:05:33.501 INFO:tasks.workunit.client.0.trial150.stdout: "waiting_on_whom": [], 2026-02-22T21:05:33.501 INFO:tasks.workunit.client.0.trial150.stdout: "schedule": "scrubbing", 2026-02-22T21:05:33.501 INFO:tasks.workunit.client.0.trial150.stdout: "is_reserving_replicas": false 2026-02-22T21:05:33.501 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:33.501 INFO:tasks.workunit.client.0.trial150.stdout: "agent_state": {} 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:( 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[query_epoch]=20 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[query_seq]=142 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[query_active]=true 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule]='scrubbing' 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule_at]='0' 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[query_target_at]='0' 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_duration]=1 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_stamp]='2026-02-20T21:05:30.705195+0000' 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_scrub]='18x90' 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[query_is_future]=false 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[query_vs_date]=false 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[query_scrub_seq]=null 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_pg_state]='active+clean' 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_state_has_scrubbing]=false 2026-02-22T21:05:33.584 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_last_duration]=1 2026-02-22T21:05:33.585 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-22T21:05:33.585 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule_at]='2026-02-22T21:05:31.978132+0000' 2026-02-22T21:05:33.585 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_is_future]=false 2026-02-22T21:05:33.585 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_vs_date]=true 2026-02-22T21:05:33.585 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_reported_epoch]=19 2026-02-22T21:05:33.585 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_seq]=135 2026-02-22T21:05:33.585 INFO:tasks.workunit.client.0.trial150.stdout:) 2026-02-22T21:05:33.595 INFO:tasks.workunit.client.0.trial150.stdout:--> loop: 1 ~ true / 142 / 135 / false / 2026-02-20T21:05:30.705195+0000 / scrubbing %%% query_active 2026-02-22T21:05:33.595 INFO:tasks.workunit.client.0.trial150.stdout:key is query_active: negation:0 # expected: true # in actual: true 2026-02-22T21:05:33.595 INFO:tasks.workunit.client.0.trial150.stdout:WaitingActive - 'query_active' actual value (true) matches expected (true) (negation: 0) 2026-02-22T21:05:33.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-22T21:05:33.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:645: TEST_dump_scrub_schedule: cond_active_dmp=(['dmp_state_has_scrubbing']='true' ['query_active']='false') 2026-02-22T21:05:33.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:645: TEST_dump_scrub_schedule: declare -A cond_active_dmp 2026-02-22T21:05:33.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:646: TEST_dump_scrub_schedule: sched_data=() 2026-02-22T21:05:33.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:647: TEST_dump_scrub_schedule: wait_any_cond 1.0 10 2026-02-20T21:05:30.705195+0000 cond_active_dmp 'WaitingActive ' sched_data 2026-02-22T21:05:33.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.0 2026-02-22T21:05:33.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-22T21:05:33.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-20T21:05:30.705195+0000 2026-02-22T21:05:33.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=cond_active_dmp 2026-02-22T21:05:33.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sched_data 2026-02-22T21:05:33.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-22T21:05:33.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-22T21:05:33.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-22T21:05:33.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-22T21:05:33.597 INFO:tasks.workunit.client.0.trial150.stdout:waiting for any condition (WaitingActive ): pg:1.0 dt:2026-02-20T21:05:30.705195+0000 (10 retries) 2026-02-22T21:05:34.247 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:05:34.256 INFO:tasks.workunit.client.0.trial150.stdout:from pg dump pg: { 2026-02-22T21:05:34.256 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_pg_state": "active+clean+scrubbing", 2026-02-22T21:05:34.257 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_state_has_scrubbing": true, 2026-02-22T21:05:34.257 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_last_duration": 1, 2026-02-22T21:05:34.257 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule": "scrubbing for 1s", 2026-02-22T21:05:34.257 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule_at": "0", 2026-02-22T21:05:34.257 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_is_future": false, 2026-02-22T21:05:34.257 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_vs_date": false, 2026-02-22T21:05:34.257 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_reported_epoch": 20, 2026-02-22T21:05:34.257 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_seq": 141 2026-02-22T21:05:34.257 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:34.257 INFO:tasks.workunit.client.0.trial150.stdout:query output: 2026-02-22T21:05:34.328 INFO:tasks.workunit.client.0.trial150.stdout: "scrubber": { 2026-02-22T21:05:34.328 INFO:tasks.workunit.client.0.trial150.stdout: "active": true, 2026-02-22T21:05:34.328 INFO:tasks.workunit.client.0.trial150.stdout: "epoch_start": "15", 2026-02-22T21:05:34.328 INFO:tasks.workunit.client.0.trial150.stdout: "start": "1:00000000::::0", 2026-02-22T21:05:34.328 INFO:tasks.workunit.client.0.trial150.stdout: "end": "MIN", 2026-02-22T21:05:34.329 INFO:tasks.workunit.client.0.trial150.stdout: "max_end": "MIN", 2026-02-22T21:05:34.329 INFO:tasks.workunit.client.0.trial150.stdout: "subset_last_update": "0'0", 2026-02-22T21:05:34.329 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T21:05:34.329 INFO:tasks.workunit.client.0.trial150.stdout: "req_scrub": false, 2026-02-22T21:05:34.329 INFO:tasks.workunit.client.0.trial150.stdout: "auto_repair": false, 2026-02-22T21:05:34.329 INFO:tasks.workunit.client.0.trial150.stdout: "check_repair": false, 2026-02-22T21:05:34.329 INFO:tasks.workunit.client.0.trial150.stdout: "deep_scrub_on_error": false, 2026-02-22T21:05:34.329 INFO:tasks.workunit.client.0.trial150.stdout: "priority": 5, 2026-02-22T21:05:34.329 INFO:tasks.workunit.client.0.trial150.stdout: "shallow_errors": 0, 2026-02-22T21:05:34.329 INFO:tasks.workunit.client.0.trial150.stdout: "deep_errors": 0, 2026-02-22T21:05:34.329 INFO:tasks.workunit.client.0.trial150.stdout: "fixed": 0, 2026-02-22T21:05:34.329 INFO:tasks.workunit.client.0.trial150.stdout: "waiting_on_whom": [], 2026-02-22T21:05:34.330 INFO:tasks.workunit.client.0.trial150.stdout: "schedule": "scrubbing", 2026-02-22T21:05:34.330 INFO:tasks.workunit.client.0.trial150.stdout: "is_reserving_replicas": false 2026-02-22T21:05:34.330 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:34.330 INFO:tasks.workunit.client.0.trial150.stdout: "agent_state": {} 2026-02-22T21:05:34.412 INFO:tasks.workunit.client.0.trial150.stdout:( 2026-02-22T21:05:34.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_epoch]=20 2026-02-22T21:05:34.413 INFO:tasks.workunit.client.0.trial150.stdout:[query_seq]=142 2026-02-22T21:05:34.413 INFO:tasks.workunit.client.0.trial150.stdout:[query_active]=true 2026-02-22T21:05:34.413 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule]='scrubbing' 2026-02-22T21:05:34.413 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule_at]='0' 2026-02-22T21:05:34.413 INFO:tasks.workunit.client.0.trial150.stdout:[query_target_at]='0' 2026-02-22T21:05:34.413 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_duration]=1 2026-02-22T21:05:34.413 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_stamp]='2026-02-20T21:05:30.705195+0000' 2026-02-22T21:05:34.413 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_scrub]='18x90' 2026-02-22T21:05:34.413 INFO:tasks.workunit.client.0.trial150.stdout:[query_is_future]=false 2026-02-22T21:05:34.413 INFO:tasks.workunit.client.0.trial150.stdout:[query_vs_date]=false 2026-02-22T21:05:34.413 INFO:tasks.workunit.client.0.trial150.stdout:[query_scrub_seq]=null 2026-02-22T21:05:34.414 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_pg_state]='active+clean+scrubbing' 2026-02-22T21:05:34.414 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_state_has_scrubbing]=true 2026-02-22T21:05:34.414 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_last_duration]=1 2026-02-22T21:05:34.414 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule]='scrubbing for 1s' 2026-02-22T21:05:34.414 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule_at]='0' 2026-02-22T21:05:34.414 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_is_future]=false 2026-02-22T21:05:34.414 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_vs_date]=false 2026-02-22T21:05:34.414 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_reported_epoch]=20 2026-02-22T21:05:34.414 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_seq]=141 2026-02-22T21:05:34.414 INFO:tasks.workunit.client.0.trial150.stdout:) 2026-02-22T21:05:34.424 INFO:tasks.workunit.client.0.trial150.stdout:--> loop: 1 ~ true / 142 / 141 / false / 2026-02-20T21:05:30.705195+0000 / scrubbing %%% dmp_state_has_scrubbing query_active 2026-02-22T21:05:34.424 INFO:tasks.workunit.client.0.trial150.stdout:key is dmp_state_has_scrubbing: negation:0 # expected: true # in actual: true 2026-02-22T21:05:34.424 INFO:tasks.workunit.client.0.trial150.stdout:WaitingActive - 'dmp_state_has_scrubbing' actual value (true) matches expected (true) (negation: 0) 2026-02-22T21:05:34.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-22T21:05:34.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:648: TEST_dump_scrub_schedule: sleep 4 2026-02-22T21:05:38.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:649: TEST_dump_scrub_schedule: perf_counters td/osd-scrub-test 3 2026-02-22T21:05:38.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-22T21:05:38.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-22T21:05:38.427 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-22T21:05:38.428 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-22T21:05:38.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:05:38.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-22T21:05:38.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:05:38.501 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:05:38.502 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:05:38.502 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:05:38.502 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:05:38.502 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:05:38.502 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:05:38.502 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:38.502 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:05:38.503 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:05:38.503 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:05:38.503 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:05:38.503 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:05:38.503 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:05:38.503 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.503 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.503 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:05:38.503 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:05:38.503 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:05:38.503 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:05:38.504 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:05:38.504 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:05:38.504 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:38.504 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:05:38.504 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:05:38.504 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:05:38.504 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:05:38.504 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:05:38.504 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:05:38.504 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.504 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.505 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:05:38.505 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:05:38.505 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:05:38.505 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:05:38.505 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:05:38.505 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:05:38.505 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:38.505 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:05:38.505 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:05:38.505 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:05:38.505 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:05:38.506 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:05:38.506 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:05:38.506 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.506 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.506 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:05:38.506 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:05:38.506 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:05:38.506 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:05:38.506 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:05:38.506 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:05:38.507 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:38.507 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:05:38.507 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:05:38.507 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:05:38.507 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:05:38.507 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:05:38.507 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:05:38.507 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.507 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.507 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:05:38.508 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:38.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:05:38.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-22T21:05:38.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:05:38.571 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:05:38.572 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:05:38.572 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:05:38.572 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:05:38.572 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:05:38.572 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:05:38.572 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:38.572 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:05:38.572 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:05:38.572 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:05:38.572 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:05:38.572 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:05:38.573 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:05:38.573 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.573 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.573 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:05:38.573 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:05:38.573 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:05:38.573 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:05:38.573 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:05:38.573 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:05:38.573 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:38.573 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:05:38.573 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:05:38.574 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 7, 2026-02-22T21:05:38.574 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:05:38.574 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:05:38.574 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:05:38.574 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.574 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.574 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:05:38.574 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:05:38.574 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:05:38.574 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:05:38.574 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:05:38.574 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:05:38.575 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:05:38.576 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:05:38.576 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:05:38.576 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:38.576 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:05:38.576 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:05:38.576 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 2, 2026-02-22T21:05:38.576 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:05:38.576 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:05:38.576 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:05:38.576 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.576 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.576 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:05:38.577 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:38.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:05:38.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-22T21:05:38.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:05:38.641 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:05:38.642 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:05:38.642 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:05:38.642 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:05:38.642 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:05:38.642 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:05:38.642 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:38.642 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:05:38.642 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:05:38.643 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:05:38.643 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:05:38.643 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:05:38.643 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:05:38.643 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.643 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.643 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:05:38.643 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:05:38.643 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:05:38.643 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:05:38.643 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:05:38.644 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:05:38.645 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:05:38.645 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:05:38.645 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:05:38.645 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:38.645 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:05:38.645 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:05:38.645 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:05:38.645 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:05:38.645 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:05:38.645 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:05:38.645 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.645 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:05:38.646 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:05:38.647 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.647 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:05:38.647 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:05:38.647 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:38.647 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Teardown Test TEST_dump_scrub_schedule ------------------ 2026-02-22T21:05:38.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_dump_scrub_schedule ------------------' 2026-02-22T21:05:38.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-22T21:05:38.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:05:38.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:05:38.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:05:38.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:05:38.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:05:38.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:05:38.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:05:38.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:05:38.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:05:38.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:05:38.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:05:38.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:05:38.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:05:38.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:05:38.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:05:38.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:05:38.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:05:38.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:05:38.757 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:05:38.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:05:38.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:05:38.759 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:05:38.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:05:38.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:38.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:38.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:05:38.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:05:38.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:05:38.773 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Complete Test TEST_dump_scrub_schedule ------------------ 2026-02-22T21:05:38.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_dump_scrub_schedule ------------------' 2026-02-22T21:05:38.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-22T21:05:38.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_interval_changes -------------------' 2026-02-22T21:05:38.774 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Prepare Test TEST_interval_changes ------------------- 2026-02-22T21:05:38.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-22T21:05:38.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-22T21:05:38.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-22T21:05:38.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:05:38.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:05:38.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:05:38.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:05:38.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:05:38.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:05:38.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:05:38.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:05:38.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:05:38.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:05:38.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:05:38.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:05:38.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:05:38.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:05:38.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:05:38.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:05:38.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:05:38.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:05:38.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:05:38.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:05:38.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:05:38.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:05:38.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:05:38.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:38.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:38.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:05:38.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:05:38.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:05:38.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-22T21:05:38.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T21:05:38.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:38.789 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:38.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.516403 2026-02-22T21:05:38.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T21:05:38.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:05:38.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T21:05:38.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-22T21:05:38.792 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Run Test TEST_interval_changes ----------------------- 2026-02-22T21:05:38.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_interval_changes -----------------------' 2026-02-22T21:05:38.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_interval_changes td/osd-scrub-test 2026-02-22T21:05:38.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:142: TEST_interval_changes: local poolname=test 2026-02-22T21:05:38.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:143: TEST_interval_changes: local OSDS=2 2026-02-22T21:05:38.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:144: TEST_interval_changes: local objects=10 2026-02-22T21:05:38.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:146: TEST_interval_changes: expr 24 '*' 60 '*' 60 2026-02-22T21:05:38.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:146: TEST_interval_changes: local day=86400 2026-02-22T21:05:38.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:147: TEST_interval_changes: expr 86400 '*' 7 2026-02-22T21:05:38.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:147: TEST_interval_changes: local week=604800 2026-02-22T21:05:38.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:148: TEST_interval_changes: local min_interval=86400 2026-02-22T21:05:38.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:149: TEST_interval_changes: local max_interval=604800 2026-02-22T21:05:38.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:150: TEST_interval_changes: local WAIT_FOR_UPDATE=15 2026-02-22T21:05:38.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:152: TEST_interval_changes: TESTDATA=testdata.516403 2026-02-22T21:05:38.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:155: TEST_interval_changes: run_mon td/osd-scrub-test a --osd_pool_default_size=2 2026-02-22T21:05:38.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-22T21:05:38.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T21:05:38.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T21:05:38.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T21:05:38.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-22T21:05:38.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=2 2026-02-22T21:05:38.816 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T21:05:38.816 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:05:38.816 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:05:38.817 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:05:38.817 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:38.817 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:38.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:05:38.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-22T21:05:38.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T21:05:38.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T21:05:38.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:05:38.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:05:38.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T21:05:38.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T21:05:38.847 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:05:38.847 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:05:38.847 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:05:38.847 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:05:38.848 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:38.848 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:38.848 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T21:05:38.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:05:38.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get fsid 2026-02-22T21:05:38.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T21:05:38.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:05:38.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:05:38.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T21:05:38.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T21:05:38.902 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:05:38.902 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:05:38.902 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:05:38.902 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:05:38.902 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:38.902 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:38.903 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T21:05:38.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:05:38.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get mon_host 2026-02-22T21:05:38.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:156: TEST_interval_changes: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-22T21:05:38.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-22T21:05:38.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T21:05:38.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T21:05:38.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T21:05:38.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-22T21:05:38.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:05:39.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T21:05:39.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:05:39.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:05:39.055 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:05:39.055 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:39.055 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:39.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:05:39.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:05:39.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-22T21:05:39.075 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: expr 2 - 1 2026-02-22T21:05:39.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: seq 0 1 2026-02-22T21:05:39.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:05:39.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:159: TEST_interval_changes: run_osd td/osd-scrub-test 0 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2026-02-22T21:05:39.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:05:39.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:05:39.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T21:05:39.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:05:39.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-22T21:05:39.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:05:39.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:05:39.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:05:39.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:05:39.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-22T21:05:39.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-22T21:05:39.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:05:39.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:05:39.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:05:39.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:05:39.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:05:39.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:05:39.082 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:05:39.082 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:39.082 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:39.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:05:39.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:05:39.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:05:39.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:05:39.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:05:39.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:05:39.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:05:39.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:05:39.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:05:39.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:05:39.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:05:39.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:05:39.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2026-02-22T21:05:39.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-22T21:05:39.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:05:39.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=97bdc0e0-cfc2-4142-ac57-93007a7a1d0e 2026-02-22T21:05:39.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 97bdc0e0-cfc2-4142-ac57-93007a7a1d0e' 2026-02-22T21:05:39.085 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 97bdc0e0-cfc2-4142-ac57-93007a7a1d0e 2026-02-22T21:05:39.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:05:39.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCjb5tpsOM2BRAAMo+DlwPmqh7ICZQmS7tDNA== 2026-02-22T21:05:39.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCjb5tpsOM2BRAAMo+DlwPmqh7ICZQmS7tDNA=="}' 2026-02-22T21:05:39.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 97bdc0e0-cfc2-4142-ac57-93007a7a1d0e -i td/osd-scrub-test/0/new.json 2026-02-22T21:05:39.182 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:05:39.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-22T21:05:39.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQCjb5tpsOM2BRAAMo+DlwPmqh7ICZQmS7tDNA== --osd-uuid 97bdc0e0-cfc2-4142-ac57-93007a7a1d0e 2026-02-22T21:05:39.207 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:39.196+0000 7f0de7852a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:39.208 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:39.200+0000 7f0de7852a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:39.209 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:39.200+0000 7f0de7852a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:39.210 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:39.200+0000 7f0de7852a40 -1 bdev(0x55eaed992c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:05:39.210 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:39.200+0000 7f0de7852a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-22T21:05:39.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-22T21:05:39.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:05:39.917 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T21:05:39.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:05:39.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:05:40.117 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T21:05:40.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T21:05:40.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2026-02-22T21:05:40.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:05:40.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:05:40.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:05:40.141 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:40.132+0000 7f80dadd3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:40.143 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:40.132+0000 7f80dadd3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:40.144 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:40.132+0000 7f80dadd3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:40.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T21:05:40.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:05:40.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T21:05:40.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:05:40.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:05:40.284 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:05:40.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:05:40.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:05:40.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:05:40.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:05:40.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:05:40.545 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:40.536+0000 7f80dadd3a40 -1 Falling back to public interface 2026-02-22T21:05:41.079 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:41.068+0000 7f80dadd3a40 -1 osd.0 0 log_to_monitors true 2026-02-22T21:05:41.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:05:41.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:05:41.446 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:05:41.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:05:41.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:05:41.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:05:41.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:05:42.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:05:42.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:05:42.607 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:05:42.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:05:42.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:05:42.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:05:42.772 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2422277227,v1:127.0.0.1:6803/2422277227] [v2:127.0.0.1:6804/2422277227,v1:127.0.0.1:6805/2422277227] exists,up 97bdc0e0-cfc2-4142-ac57-93007a7a1d0e 2026-02-22T21:05:42.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:05:42.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:05:42.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:05:42.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:157: TEST_interval_changes: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:05:42.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:159: TEST_interval_changes: run_osd td/osd-scrub-test 1 --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2026-02-22T21:05:42.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:05:42.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:05:42.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:05:42.775 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:05:42.775 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:42.775 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:42.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:05:42.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:05:42.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:05:42.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:05:42.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:05:42.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:05:42.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:05:42.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:05:42.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:05:42.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:05:42.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:05:42.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:05:42.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0' 2026-02-22T21:05:42.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-22T21:05:42.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:05:42.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5e230a49-5c5e-4b21-a7ee-f106700eb9e2 2026-02-22T21:05:42.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5e230a49-5c5e-4b21-a7ee-f106700eb9e2' 2026-02-22T21:05:42.777 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 5e230a49-5c5e-4b21-a7ee-f106700eb9e2 2026-02-22T21:05:42.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:05:42.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCmb5tpLTLBLhAA0iaW/mvSLs2Q+spjEhEdCQ== 2026-02-22T21:05:42.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCmb5tpLTLBLhAA0iaW/mvSLs2Q+spjEhEdCQ=="}' 2026-02-22T21:05:42.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5e230a49-5c5e-4b21-a7ee-f106700eb9e2 -i td/osd-scrub-test/1/new.json 2026-02-22T21:05:42.951 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:05:42.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-22T21:05:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 --mkfs --key AQCmb5tpLTLBLhAA0iaW/mvSLs2Q+spjEhEdCQ== --osd-uuid 5e230a49-5c5e-4b21-a7ee-f106700eb9e2 2026-02-22T21:05:42.977 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:42.968+0000 7f3985e78a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:42.979 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:42.968+0000 7f3985e78a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:42.980 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:42.972+0000 7f3985e78a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:42.981 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:42.972+0000 7f3985e78a40 -1 bdev(0x560dae0aec00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:05:42.981 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:42.972+0000 7f3985e78a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-22T21:05:43.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-22T21:05:43.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:05:43.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:05:43.609 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T21:05:43.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:05:43.814 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T21:05:43.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T21:05:43.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_scrub_min_interval=86400 --osd_scrub_max_interval=604800 --osd_scrub_interval_randomize_ratio=0 2026-02-22T21:05:43.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:05:43.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:05:43.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:05:43.830 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:43.820+0000 7fd11ca4fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:43.832 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:43.824+0000 7fd11ca4fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:43.833 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:43.824+0000 7fd11ca4fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:05:43.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T21:05:43.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:05:43.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T21:05:43.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:05:43.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:05:43.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:05:43.974 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:05:43.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:05:43.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:05:43.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:05:44.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:05:44.206 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:44.196+0000 7fd11ca4fa40 -1 Falling back to public interface 2026-02-22T21:05:44.782 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:05:44.772+0000 7fd11ca4fa40 -1 osd.1 0 log_to_monitors true 2026-02-22T21:05:45.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:05:45.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:05:45.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:05:45.139 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:05:45.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:05:45.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:05:45.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:05:46.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:05:46.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:05:46.302 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:05:46.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:05:46.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:05:46.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:05:46.453 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4091147009,v1:127.0.0.1:6811/4091147009] [v2:127.0.0.1:6812/4091147009,v1:127.0.0.1:6813/4091147009] exists,up 5e230a49-5c5e-4b21-a7ee-f106700eb9e2 2026-02-22T21:05:46.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:05:46.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:05:46.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:05:46.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:163: TEST_interval_changes: create_pool test 1 1 2026-02-22T21:05:46.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-22T21:05:46.659 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T21:05:46.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T21:05:47.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:164: TEST_interval_changes: wait_for_clean 2026-02-22T21:05:47.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:05:47.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:05:47.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:05:47.672 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:05:47.672 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:05:47.673 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:05:47.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:05:47.673 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:05:47.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:05:47.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:05:47.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:05:47.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:05:47.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:05:47.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:05:47.776 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:05:47.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:05:47.941 INFO:tasks.workunit.client.0.trial150.stderr:1' 2026-02-22T21:05:47.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:05:47.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:05:47.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:05:48.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-22T21:05:48.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-22T21:05:48.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-22T21:05:48.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:05:48.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:05:48.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-22T21:05:48.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-22T21:05:48.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-42949672964' 2026-02-22T21:05:48.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:05:48.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-22T21:05:48.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:05:48.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:05:48.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-22T21:05:48.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:05:48.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-22T21:05:48.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-22T21:05:48.091 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836487 2026-02-22T21:05:48.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:05:48.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T21:05:48.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:05:49.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:05:49.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:05:49.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T21:05:49.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:05:50.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-22T21:05:50.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:05:50.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836487 2026-02-22T21:05:50.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:05:50.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-22T21:05:50.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:05:50.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:05:50.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672964 2026-02-22T21:05:50.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:05:50.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672964 2026-02-22T21:05:50.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-22T21:05:50.585 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672964 2026-02-22T21:05:50.585 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:05:50.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672965 -lt 42949672964 2026-02-22T21:05:50.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:05:50.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:05:50.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:05:50.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:05:50.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:05:50.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:05:50.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:05:50.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:05:50.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:05:50.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:05:50.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:05:51.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:05:51.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:05:51.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:05:51.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:05:51.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:05:51.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:05:51.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:05:51.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:165: TEST_interval_changes: ceph osd dump 2026-02-22T21:05:51.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:165: TEST_interval_changes: grep '^pool.*['\'']test['\'']' 2026-02-22T21:05:51.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:165: TEST_interval_changes: awk '{ print $2 }' 2026-02-22T21:05:51.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:165: TEST_interval_changes: local poolid=1 2026-02-22T21:05:51.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:167: TEST_interval_changes: dd if=/dev/urandom of=testdata.516403 bs=1032 count=1 2026-02-22T21:05:51.446 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T21:05:51.446 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T21:05:51.446 INFO:tasks.workunit.client.0.trial150.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.5647e-05 s, 29.0 MB/s 2026-02-22T21:05:51.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: seq 1 10 2026-02-22T21:05:51.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-22T21:05:51.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj1 testdata.516403 2026-02-22T21:05:51.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-22T21:05:51.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj2 testdata.516403 2026-02-22T21:05:51.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-22T21:05:51.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj3 testdata.516403 2026-02-22T21:05:51.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-22T21:05:51.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj4 testdata.516403 2026-02-22T21:05:51.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-22T21:05:51.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj5 testdata.516403 2026-02-22T21:05:51.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-22T21:05:51.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj6 testdata.516403 2026-02-22T21:05:51.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-22T21:05:51.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj7 testdata.516403 2026-02-22T21:05:51.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-22T21:05:51.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj8 testdata.516403 2026-02-22T21:05:51.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-22T21:05:51.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj9 testdata.516403 2026-02-22T21:05:51.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:168: TEST_interval_changes: for i in `seq 1 $objects` 2026-02-22T21:05:51.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:170: TEST_interval_changes: rados -p test put obj10 testdata.516403 2026-02-22T21:05:51.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:172: TEST_interval_changes: rm -f testdata.516403 2026-02-22T21:05:51.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:174: TEST_interval_changes: get_primary test obj1 2026-02-22T21:05:51.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-22T21:05:51.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T21:05:51.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-22T21:05:51.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T21:05:51.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:174: TEST_interval_changes: local primary=1 2026-02-22T21:05:51.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:177: TEST_interval_changes: check_dump_scrubs 1 '1 day' 2026-02-22T21:05:51.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-22T21:05:51.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=1 day' 2026-02-22T21:05:51.833 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-22T21:05:51.833 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T21:05:51.833 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T21:05:51.833 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:05:51.833 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:51.833 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:51.833 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-osd.1.asok 2026-02-22T21:05:51.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-22T21:05:51.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.516403/ceph-osd.1.asok dump_scrubs 2026-02-22T21:05:51.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-22T21:05:51.893 INFO:tasks.workunit.client.0.trial150.stderr: { 2026-02-22T21:05:51.894 INFO:tasks.workunit.client.0.trial150.stderr: "pgid": "1.0", 2026-02-22T21:05:51.894 INFO:tasks.workunit.client.0.trial150.stderr: "sched_time": "2026-02-23T21:05:46.654165+0000", 2026-02-22T21:05:51.894 INFO:tasks.workunit.client.0.trial150.stderr: "orig_sched_time": "2026-02-23T21:05:46.654165+0000", 2026-02-22T21:05:51.894 INFO:tasks.workunit.client.0.trial150.stderr: "forced": false, 2026-02-22T21:05:51.894 INFO:tasks.workunit.client.0.trial150.stderr: "level": "shallow", 2026-02-22T21:05:51.894 INFO:tasks.workunit.client.0.trial150.stderr: "urgency": "periodic-regular", 2026-02-22T21:05:51.894 INFO:tasks.workunit.client.0.trial150.stderr: "eligible": false, 2026-02-22T21:05:51.894 INFO:tasks.workunit.client.0.trial150.stderr: "last_issue": "ok" 2026-02-22T21:05:51.894 INFO:tasks.workunit.client.0.trial150.stderr: }, 2026-02-22T21:05:51.894 INFO:tasks.workunit.client.0.trial150.stderr: { 2026-02-22T21:05:51.895 INFO:tasks.workunit.client.0.trial150.stderr: "pgid": "1.0", 2026-02-22T21:05:51.895 INFO:tasks.workunit.client.0.trial150.stderr: "sched_time": "2026-03-03T10:57:41.704394+0000", 2026-02-22T21:05:51.895 INFO:tasks.workunit.client.0.trial150.stderr: "orig_sched_time": "2026-03-03T10:57:41.704394+0000", 2026-02-22T21:05:51.895 INFO:tasks.workunit.client.0.trial150.stderr: "forced": false, 2026-02-22T21:05:51.895 INFO:tasks.workunit.client.0.trial150.stderr: "level": "deep", 2026-02-22T21:05:51.895 INFO:tasks.workunit.client.0.trial150.stderr: "urgency": "periodic-regular", 2026-02-22T21:05:51.895 INFO:tasks.workunit.client.0.trial150.stderr: "eligible": false, 2026-02-22T21:05:51.895 INFO:tasks.workunit.client.0.trial150.stderr: "last_issue": "ok" 2026-02-22T21:05:51.895 INFO:tasks.workunit.client.0.trial150.stderr: } 2026-02-22T21:05:51.895 INFO:tasks.workunit.client.0.trial150.stderr:]' 2026-02-22T21:05:51.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-23T21:05:46.654165+0000",' '"orig_sched_time":' '"2026-02-23T21:05:46.654165+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-03-03T10:57:41.704394+0000",' '"orig_sched_time":' '"2026-03-03T10:57:41.704394+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-22T21:05:51.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-22T21:05:51.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-23T21:05:46.654165+0000"' 2026-02-22T21:05:51.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-23T21:05:46.654165+0000 2026-02-22T21:05:51.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-23T21:05:46.654165+0000 2026-02-22T21:05:51.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-22T21:05:51.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 1 day' 2026-02-22T21:05:51.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-23 = 2026-02-23 2026-02-22T21:05:51.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:180: TEST_interval_changes: get_asok_path osd.1 2026-02-22T21:05:51.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T21:05:51.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T21:05:51.909 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:05:51.909 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:05:51.910 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:05:51.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-osd.1.asok 2026-02-22T21:05:51.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:180: TEST_interval_changes: expr 86400 '*' 2 2026-02-22T21:05:51.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:180: TEST_interval_changes: CEPH_ARGS= 2026-02-22T21:05:51.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:180: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.516403/ceph-osd.1.asok config set osd_scrub_min_interval 172800 2026-02-22T21:05:51.964 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:05:51.964 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_min_interval = '' " 2026-02-22T21:05:51.964 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:05:51.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:181: TEST_interval_changes: sleep 15 2026-02-22T21:06:06.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:182: TEST_interval_changes: check_dump_scrubs 1 '2 days' 2026-02-22T21:06:06.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-22T21:06:06.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=2 days' 2026-02-22T21:06:06.972 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-22T21:06:06.972 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T21:06:06.972 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T21:06:06.973 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:06:06.973 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:06:06.973 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:06:06.973 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-osd.1.asok 2026-02-22T21:06:06.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-22T21:06:06.974 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.516403/ceph-osd.1.asok dump_scrubs 2026-02-22T21:06:07.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-22T21:06:07.032 INFO:tasks.workunit.client.0.trial150.stderr: { 2026-02-22T21:06:07.033 INFO:tasks.workunit.client.0.trial150.stderr: "pgid": "1.0", 2026-02-22T21:06:07.033 INFO:tasks.workunit.client.0.trial150.stderr: "sched_time": "2026-02-24T21:05:46.654165+0000", 2026-02-22T21:06:07.033 INFO:tasks.workunit.client.0.trial150.stderr: "orig_sched_time": "2026-02-24T21:05:46.654165+0000", 2026-02-22T21:06:07.033 INFO:tasks.workunit.client.0.trial150.stderr: "forced": false, 2026-02-22T21:06:07.033 INFO:tasks.workunit.client.0.trial150.stderr: "level": "shallow", 2026-02-22T21:06:07.033 INFO:tasks.workunit.client.0.trial150.stderr: "urgency": "periodic-regular", 2026-02-22T21:06:07.033 INFO:tasks.workunit.client.0.trial150.stderr: "eligible": false, 2026-02-22T21:06:07.033 INFO:tasks.workunit.client.0.trial150.stderr: "last_issue": "ok" 2026-02-22T21:06:07.033 INFO:tasks.workunit.client.0.trial150.stderr: }, 2026-02-22T21:06:07.034 INFO:tasks.workunit.client.0.trial150.stderr: { 2026-02-22T21:06:07.034 INFO:tasks.workunit.client.0.trial150.stderr: "pgid": "1.0", 2026-02-22T21:06:07.034 INFO:tasks.workunit.client.0.trial150.stderr: "sched_time": "2026-03-01T05:16:48.187148+0000", 2026-02-22T21:06:07.034 INFO:tasks.workunit.client.0.trial150.stderr: "orig_sched_time": "2026-03-01T05:16:48.187148+0000", 2026-02-22T21:06:07.034 INFO:tasks.workunit.client.0.trial150.stderr: "forced": false, 2026-02-22T21:06:07.034 INFO:tasks.workunit.client.0.trial150.stderr: "level": "deep", 2026-02-22T21:06:07.034 INFO:tasks.workunit.client.0.trial150.stderr: "urgency": "periodic-regular", 2026-02-22T21:06:07.034 INFO:tasks.workunit.client.0.trial150.stderr: "eligible": false, 2026-02-22T21:06:07.034 INFO:tasks.workunit.client.0.trial150.stderr: "last_issue": "ok" 2026-02-22T21:06:07.034 INFO:tasks.workunit.client.0.trial150.stderr: } 2026-02-22T21:06:07.034 INFO:tasks.workunit.client.0.trial150.stderr:]' 2026-02-22T21:06:07.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-22T21:06:07.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-24T21:05:46.654165+0000",' '"orig_sched_time":' '"2026-02-24T21:05:46.654165+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-03-01T05:16:48.187148+0000",' '"orig_sched_time":' '"2026-03-01T05:16:48.187148+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-22T21:06:07.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-24T21:05:46.654165+0000"' 2026-02-22T21:06:07.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-24T21:05:46.654165+0000 2026-02-22T21:06:07.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-24T21:05:46.654165+0000 2026-02-22T21:06:07.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-22T21:06:07.047 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2026-02-22T21:06:07.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-24 = 2026-02-24 2026-02-22T21:06:07.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:185: TEST_interval_changes: get_asok_path osd.1 2026-02-22T21:06:07.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T21:06:07.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T21:06:07.049 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:06:07.049 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:06:07.049 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:06:07.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-osd.1.asok 2026-02-22T21:06:07.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:185: TEST_interval_changes: expr 604800 '*' 2 2026-02-22T21:06:07.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:185: TEST_interval_changes: CEPH_ARGS= 2026-02-22T21:06:07.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:185: TEST_interval_changes: ceph --admin-daemon /tmp/ceph-asok.516403/ceph-osd.1.asok config set osd_scrub_max_interval 1209600 2026-02-22T21:06:07.104 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:06:07.105 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_max_interval = '' " 2026-02-22T21:06:07.105 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:06:07.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:186: TEST_interval_changes: sleep 15 2026-02-22T21:06:22.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:187: TEST_interval_changes: check_dump_scrubs 1 '2 days' 2026-02-22T21:06:22.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-22T21:06:22.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=2 days' 2026-02-22T21:06:22.113 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-22T21:06:22.113 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T21:06:22.113 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T21:06:22.114 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:06:22.114 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:06:22.114 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:06:22.114 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-osd.1.asok 2026-02-22T21:06:22.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-22T21:06:22.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.516403/ceph-osd.1.asok dump_scrubs 2026-02-22T21:06:22.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-22T21:06:22.173 INFO:tasks.workunit.client.0.trial150.stderr: { 2026-02-22T21:06:22.174 INFO:tasks.workunit.client.0.trial150.stderr: "pgid": "1.0", 2026-02-22T21:06:22.174 INFO:tasks.workunit.client.0.trial150.stderr: "sched_time": "2026-02-24T21:05:46.654165+0000", 2026-02-22T21:06:22.174 INFO:tasks.workunit.client.0.trial150.stderr: "orig_sched_time": "2026-02-24T21:05:46.654165+0000", 2026-02-22T21:06:22.174 INFO:tasks.workunit.client.0.trial150.stderr: "forced": false, 2026-02-22T21:06:22.174 INFO:tasks.workunit.client.0.trial150.stderr: "level": "shallow", 2026-02-22T21:06:22.174 INFO:tasks.workunit.client.0.trial150.stderr: "urgency": "periodic-regular", 2026-02-22T21:06:22.174 INFO:tasks.workunit.client.0.trial150.stderr: "eligible": false, 2026-02-22T21:06:22.174 INFO:tasks.workunit.client.0.trial150.stderr: "last_issue": "ok" 2026-02-22T21:06:22.174 INFO:tasks.workunit.client.0.trial150.stderr: }, 2026-02-22T21:06:22.174 INFO:tasks.workunit.client.0.trial150.stderr: { 2026-02-22T21:06:22.174 INFO:tasks.workunit.client.0.trial150.stderr: "pgid": "1.0", 2026-02-22T21:06:22.175 INFO:tasks.workunit.client.0.trial150.stderr: "sched_time": "2026-03-03T01:24:38.910833+0000", 2026-02-22T21:06:22.175 INFO:tasks.workunit.client.0.trial150.stderr: "orig_sched_time": "2026-03-03T01:24:38.910833+0000", 2026-02-22T21:06:22.175 INFO:tasks.workunit.client.0.trial150.stderr: "forced": false, 2026-02-22T21:06:22.175 INFO:tasks.workunit.client.0.trial150.stderr: "level": "deep", 2026-02-22T21:06:22.175 INFO:tasks.workunit.client.0.trial150.stderr: "urgency": "periodic-regular", 2026-02-22T21:06:22.175 INFO:tasks.workunit.client.0.trial150.stderr: "eligible": false, 2026-02-22T21:06:22.175 INFO:tasks.workunit.client.0.trial150.stderr: "last_issue": "ok" 2026-02-22T21:06:22.175 INFO:tasks.workunit.client.0.trial150.stderr: } 2026-02-22T21:06:22.175 INFO:tasks.workunit.client.0.trial150.stderr:]' 2026-02-22T21:06:22.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-22T21:06:22.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-24T21:05:46.654165+0000",' '"orig_sched_time":' '"2026-02-24T21:05:46.654165+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-03-03T01:24:38.910833+0000",' '"orig_sched_time":' '"2026-03-03T01:24:38.910833+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-22T21:06:22.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-24T21:05:46.654165+0000"' 2026-02-22T21:06:22.184 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-24T21:05:46.654165+0000 2026-02-22T21:06:22.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-24T21:05:46.654165+0000 2026-02-22T21:06:22.185 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-22T21:06:22.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 2 days' 2026-02-22T21:06:22.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-24 = 2026-02-24 2026-02-22T21:06:22.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:190: TEST_interval_changes: expr 86400 '*' 3 2026-02-22T21:06:22.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:190: TEST_interval_changes: ceph osd pool set test scrub_min_interval 259200 2026-02-22T21:06:22.391 INFO:tasks.workunit.client.0.trial150.stderr:set pool 1 scrub_min_interval to 259200 2026-02-22T21:06:22.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:191: TEST_interval_changes: sleep 15 2026-02-22T21:06:37.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:192: TEST_interval_changes: check_dump_scrubs 1 '3 days' 2026-02-22T21:06:37.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-22T21:06:37.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=3 days' 2026-02-22T21:06:37.406 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-22T21:06:37.406 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T21:06:37.407 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T21:06:37.407 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:06:37.407 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:06:37.407 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:06:37.408 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-osd.1.asok 2026-02-22T21:06:37.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-22T21:06:37.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.516403/ceph-osd.1.asok dump_scrubs 2026-02-22T21:06:37.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-22T21:06:37.468 INFO:tasks.workunit.client.0.trial150.stderr: { 2026-02-22T21:06:37.468 INFO:tasks.workunit.client.0.trial150.stderr: "pgid": "1.0", 2026-02-22T21:06:37.468 INFO:tasks.workunit.client.0.trial150.stderr: "sched_time": "2026-02-25T21:05:46.654165+0000", 2026-02-22T21:06:37.468 INFO:tasks.workunit.client.0.trial150.stderr: "orig_sched_time": "2026-02-25T21:05:46.654165+0000", 2026-02-22T21:06:37.468 INFO:tasks.workunit.client.0.trial150.stderr: "forced": false, 2026-02-22T21:06:37.468 INFO:tasks.workunit.client.0.trial150.stderr: "level": "shallow", 2026-02-22T21:06:37.468 INFO:tasks.workunit.client.0.trial150.stderr: "urgency": "periodic-regular", 2026-02-22T21:06:37.468 INFO:tasks.workunit.client.0.trial150.stderr: "eligible": false, 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr: "last_issue": "ok" 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr: }, 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr: { 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr: "pgid": "1.0", 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr: "sched_time": "2026-03-01T11:49:17.961032+0000", 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr: "orig_sched_time": "2026-03-01T11:49:17.961032+0000", 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr: "forced": false, 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr: "level": "deep", 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr: "urgency": "periodic-regular", 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr: "eligible": false, 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr: "last_issue": "ok" 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr: } 2026-02-22T21:06:37.469 INFO:tasks.workunit.client.0.trial150.stderr:]' 2026-02-22T21:06:37.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-22T21:06:37.470 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-25T21:05:46.654165+0000",' '"orig_sched_time":' '"2026-02-25T21:05:46.654165+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-03-01T11:49:17.961032+0000",' '"orig_sched_time":' '"2026-03-01T11:49:17.961032+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-22T21:06:37.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-25T21:05:46.654165+0000"' 2026-02-22T21:06:37.479 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-25T21:05:46.654165+0000 2026-02-22T21:06:37.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-25T21:05:46.654165+0000 2026-02-22T21:06:37.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-22T21:06:37.482 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2026-02-22T21:06:37.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-25 = 2026-02-25 2026-02-22T21:06:37.483 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:195: TEST_interval_changes: expr 604800 '*' 3 2026-02-22T21:06:37.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:195: TEST_interval_changes: ceph osd pool set test scrub_max_interval 1814400 2026-02-22T21:06:37.706 INFO:tasks.workunit.client.0.trial150.stderr:set pool 1 scrub_max_interval to 1814400 2026-02-22T21:06:37.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:196: TEST_interval_changes: sleep 15 2026-02-22T21:06:52.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:197: TEST_interval_changes: check_dump_scrubs 1 '3 days' 2026-02-22T21:06:52.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:132: check_dump_scrubs: local primary=1 2026-02-22T21:06:52.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:133: check_dump_scrubs: local 'sched_time_check=3 days' 2026-02-22T21:06:52.720 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: get_asok_path osd.1 2026-02-22T21:06:52.720 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-22T21:06:52.720 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-22T21:06:52.721 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:06:52.721 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:06:52.721 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:06:52.721 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-osd.1.asok 2026-02-22T21:06:52.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: CEPH_ARGS= 2026-02-22T21:06:52.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: ceph --admin-daemon /tmp/ceph-asok.516403/ceph-osd.1.asok dump_scrubs 2026-02-22T21:06:52.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:135: check_dump_scrubs: DS='[ 2026-02-22T21:06:52.783 INFO:tasks.workunit.client.0.trial150.stderr: { 2026-02-22T21:06:52.783 INFO:tasks.workunit.client.0.trial150.stderr: "pgid": "1.0", 2026-02-22T21:06:52.783 INFO:tasks.workunit.client.0.trial150.stderr: "sched_time": "2026-02-25T21:05:46.654165+0000", 2026-02-22T21:06:52.783 INFO:tasks.workunit.client.0.trial150.stderr: "orig_sched_time": "2026-02-25T21:05:46.654165+0000", 2026-02-22T21:06:52.783 INFO:tasks.workunit.client.0.trial150.stderr: "forced": false, 2026-02-22T21:06:52.783 INFO:tasks.workunit.client.0.trial150.stderr: "level": "shallow", 2026-02-22T21:06:52.783 INFO:tasks.workunit.client.0.trial150.stderr: "urgency": "periodic-regular", 2026-02-22T21:06:52.784 INFO:tasks.workunit.client.0.trial150.stderr: "eligible": false, 2026-02-22T21:06:52.784 INFO:tasks.workunit.client.0.trial150.stderr: "last_issue": "ok" 2026-02-22T21:06:52.784 INFO:tasks.workunit.client.0.trial150.stderr: }, 2026-02-22T21:06:52.784 INFO:tasks.workunit.client.0.trial150.stderr: { 2026-02-22T21:06:52.784 INFO:tasks.workunit.client.0.trial150.stderr: "pgid": "1.0", 2026-02-22T21:06:52.784 INFO:tasks.workunit.client.0.trial150.stderr: "sched_time": "2026-03-01T16:53:38.392917+0000", 2026-02-22T21:06:52.784 INFO:tasks.workunit.client.0.trial150.stderr: "orig_sched_time": "2026-03-01T16:53:38.392917+0000", 2026-02-22T21:06:52.784 INFO:tasks.workunit.client.0.trial150.stderr: "forced": false, 2026-02-22T21:06:52.784 INFO:tasks.workunit.client.0.trial150.stderr: "level": "deep", 2026-02-22T21:06:52.784 INFO:tasks.workunit.client.0.trial150.stderr: "urgency": "periodic-regular", 2026-02-22T21:06:52.784 INFO:tasks.workunit.client.0.trial150.stderr: "eligible": false, 2026-02-22T21:06:52.784 INFO:tasks.workunit.client.0.trial150.stderr: "last_issue": "ok" 2026-02-22T21:06:52.785 INFO:tasks.workunit.client.0.trial150.stderr: } 2026-02-22T21:06:52.785 INFO:tasks.workunit.client.0.trial150.stderr:]' 2026-02-22T21:06:52.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: jq '.[0].sched_time' 2026-02-22T21:06:52.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: echo '[' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-02-25T21:05:46.654165+0000",' '"orig_sched_time":' '"2026-02-25T21:05:46.654165+0000",' '"forced":' false, '"level":' '"shallow",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '},' '{' '"pgid":' '"1.0",' '"sched_time":' '"2026-03-01T16:53:38.392917+0000",' '"orig_sched_time":' '"2026-03-01T16:53:38.392917+0000",' '"forced":' false, '"level":' '"deep",' '"urgency":' '"periodic-regular",' '"eligible":' false, '"last_issue":' '"ok"' '}' ']' 2026-02-22T21:06:52.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: eval 'SCHED_TIME="2026-02-25T21:05:46.654165+0000"' 2026-02-22T21:06:52.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:137: check_dump_scrubs: SCHED_TIME=2026-02-25T21:05:46.654165+0000 2026-02-22T21:06:52.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: echo 2026-02-25T21:05:46.654165+0000 2026-02-22T21:06:52.794 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: sed 's/\([0-9]*-[0-9]*-[0-9]*\).*/\1/' 2026-02-22T21:06:52.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: date +%Y-%m-%d -d 'now + 3 days' 2026-02-22T21:06:52.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:138: check_dump_scrubs: test 2026-02-25 = 2026-02-25 2026-02-22T21:06:52.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:198: TEST_interval_changes: perf_counters td/osd-scrub-test 2 2026-02-22T21:06:52.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-22T21:06:52.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=2 2026-02-22T21:06:52.798 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 2 - 1 2026-02-22T21:06:52.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 1 2026-02-22T21:06:52.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:06:52.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-22T21:06:52.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:06:52.870 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:06:52.870 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:06:52.870 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:06:52.870 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:06:52.870 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:06:52.870 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:06:52.870 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:06:52.870 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:06:52.871 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:06:52.871 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:06:52.871 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:06:52.871 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:06:52.871 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:06:52.871 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.871 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.871 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:06:52.871 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:06:52.871 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:06:52.871 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:06:52.872 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.873 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:06:52.874 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:06:52.875 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.875 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.875 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:06:52.875 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:06:52.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:06:52.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-22T21:06:52.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:06:52.943 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:06:52.943 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:06:52.943 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:06:52.943 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:06:52.943 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:06:52.943 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:06:52.943 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:06:52.943 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:06:52.943 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:06:52.943 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:06:52.944 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Teardown Test TEST_interval_changes ------------------ 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_interval_changes ------------------' 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:06:52.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:06:53.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:06:53.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:06:53.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:06:53.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:06:53.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:06:53.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:06:53.057 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:06:53.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:06:53.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:06:53.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:06:53.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:06:53.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:06:53.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:06:53.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:06:53.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:06:53.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:06:53.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:06:53.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:06:53.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:06:53.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:06:53.072 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Complete Test TEST_interval_changes ------------------ 2026-02-22T21:06:53.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_interval_changes ------------------' 2026-02-22T21:06:53.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-22T21:06:53.073 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Prepare Test TEST_just_deep_scrubs ------------------- 2026-02-22T21:06:53.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_just_deep_scrubs -------------------' 2026-02-22T21:06:53.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-22T21:06:53.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-22T21:06:53.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-22T21:06:53.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:06:53.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:06:53.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:06:53.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:06:53.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:06:53.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:06:53.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:06:53.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:06:53.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:06:53.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:06:53.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:06:53.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:06:53.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:06:53.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:06:53.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:06:53.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:06:53.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:06:53.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:06:53.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:06:53.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:06:53.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:06:53.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:06:53.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:06:53.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:06:53.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:06:53.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:06:53.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:06:53.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:06:53.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-22T21:06:53.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T21:06:53.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:06:53.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:06:53.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.516403 2026-02-22T21:06:53.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T21:06:53.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:06:53.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T21:06:53.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Run Test TEST_just_deep_scrubs ----------------------- 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_just_deep_scrubs -----------------------' 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_just_deep_scrubs td/osd-scrub-test 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:462: TEST_just_deep_scrubs: local dir=td/osd-scrub-test 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: cluster_conf=(['osds_num']='3' ['pgs_in_pool']='4' ['pool_name']='test') 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:463: TEST_just_deep_scrubs: local -A cluster_conf 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:469: TEST_just_deep_scrubs: standard_scrub_cluster td/osd-scrub-test cluster_conf 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:229: standard_scrub_cluster: local dir=td/osd-scrub-test 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:230: standard_scrub_cluster: local -n args=cluster_conf 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:232: standard_scrub_cluster: local OSDS=3 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:233: standard_scrub_cluster: local pg_num=4 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:235: standard_scrub_cluster: local poolname=test 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:236: standard_scrub_cluster: args['pool_name']=test 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:238: standard_scrub_cluster: local pool_default_size=3 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:239: standard_scrub_cluster: args['pool_default_size']=3 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:241: standard_scrub_cluster: local extra_pars= 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:242: standard_scrub_cluster: local debug_msg=dbg 2026-02-22T21:06:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:245: standard_scrub_cluster: local saved_echo_flag=x 2026-02-22T21:06:53.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:246: standard_scrub_cluster: set +x 2026-02-22T21:06:53.374 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 2065d4d3-6e7c-4dc0-a99e-17df4cfc2a57 2026-02-22T21:06:53.473 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:06:53.502 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:53.492+0000 7f8789c7ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:06:53.503 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:53.492+0000 7f8789c7ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:06:53.504 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:53.496+0000 7f8789c7ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:06:53.505 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:53.496+0000 7f8789c7ea40 -1 bdev(0x55a31a39ec00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:06:53.505 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:53.496+0000 7f8789c7ea40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-22T21:06:54.289 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T21:06:54.489 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T21:06:54.518 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:54.508+0000 7fb6192d9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:06:54.520 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:54.508+0000 7fb6192d9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:06:54.521 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:54.512+0000 7fb6192d9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:06:54.650 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:06:54.902 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:54.892+0000 7fb6192d9a40 -1 Falling back to public interface 2026-02-22T21:06:55.514 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:55.504+0000 7fb6192d9a40 -1 osd.0 0 log_to_monitors true 2026-02-22T21:06:55.808 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:06:56.966 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:06:57.121 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1770906999,v1:127.0.0.1:6803/1770906999] [v2:127.0.0.1:6804/1770906999,v1:127.0.0.1:6805/1770906999] exists,up 2065d4d3-6e7c-4dc0-a99e-17df4cfc2a57 2026-02-22T21:06:57.126 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 92453da9-5d8d-46f4-8ef9-5f1c9193152d 2026-02-22T21:06:57.303 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:06:57.332 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:57.324+0000 7fe8e1af9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:06:57.334 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:57.324+0000 7fe8e1af9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:06:57.335 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:57.324+0000 7fe8e1af9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:06:57.335 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:57.324+0000 7fe8e1af9a40 -1 bdev(0x55df4139ac00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:06:57.335 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:57.324+0000 7fe8e1af9a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-22T21:06:57.993 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T21:06:58.193 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T21:06:58.213 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:58.204+0000 7fe048877a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:06:58.215 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:58.204+0000 7fe048877a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:06:58.215 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:58.204+0000 7fe048877a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:06:58.350 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:06:58.638 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:58.628+0000 7fe048877a40 -1 Falling back to public interface 2026-02-22T21:06:59.230 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:06:59.220+0000 7fe048877a40 -1 osd.1 0 log_to_monitors true 2026-02-22T21:06:59.510 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:07:00.665 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:07:00.826 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1999021681,v1:127.0.0.1:6811/1999021681] [v2:127.0.0.1:6812/1999021681,v1:127.0.0.1:6813/1999021681] exists,up 92453da9-5d8d-46f4-8ef9-5f1c9193152d 2026-02-22T21:07:00.830 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 078355dc-a9d3-4034-8c7d-2656b51c0387 2026-02-22T21:07:00.995 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:07:01.024 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:01.012+0000 7f8c92b29a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:01.026 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:01.016+0000 7f8c92b29a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:01.027 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:01.016+0000 7f8c92b29a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:01.028 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:01.016+0000 7f8c92b29a40 -1 bdev(0x56264936ac00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:07:01.028 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:01.016+0000 7f8c92b29a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-22T21:07:01.601 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T21:07:01.803 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T21:07:01.822 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:01.812+0000 7f82242daa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:01.823 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:01.812+0000 7f82242daa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:01.824 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:01.812+0000 7f82242daa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:01.961 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:07:02.206 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:02.196+0000 7f82242daa40 -1 Falling back to public interface 2026-02-22T21:07:02.834 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:02.824+0000 7f82242daa40 -1 osd.2 0 log_to_monitors true 2026-02-22T21:07:03.118 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:07:04.274 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:07:04.429 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/505009117,v1:127.0.0.1:6819/505009117] [v2:127.0.0.1:6820/505009117,v1:127.0.0.1:6821/505009117] exists,up 078355dc-a9d3-4034-8c7d-2656b51c0387 2026-02-22T21:07:04.622 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T21:07:06.128 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836491 2026-02-22T21:07:08.618 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705671 2026-02-22T21:07:08.783 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542148 2026-02-22T21:07:10.652 INFO:tasks.workunit.client.0.trial150.stdout:standard_scrub_cluster: dbg: test pool is test 1 2026-02-22T21:07:10.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:470: TEST_just_deep_scrubs: local poolid=1 2026-02-22T21:07:10.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:471: TEST_just_deep_scrubs: local poolname=test 2026-02-22T21:07:10.653 INFO:tasks.workunit.client.0.trial150.stdout:Pool: test : 1 2026-02-22T21:07:10.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:472: TEST_just_deep_scrubs: echo 'Pool: test : 1' 2026-02-22T21:07:10.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:474: TEST_just_deep_scrubs: TESTDATA=testdata.516403 2026-02-22T21:07:10.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:475: TEST_just_deep_scrubs: local objects=90 2026-02-22T21:07:10.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:476: TEST_just_deep_scrubs: dd if=/dev/urandom of=testdata.516403 bs=1032 count=1 2026-02-22T21:07:10.653 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T21:07:10.653 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T21:07:10.653 INFO:tasks.workunit.client.0.trial150.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.7209e-05 s, 27.7 MB/s 2026-02-22T21:07:10.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: seq 1 90 2026-02-22T21:07:10.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj1 testdata.516403 2026-02-22T21:07:10.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj2 testdata.516403 2026-02-22T21:07:10.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj3 testdata.516403 2026-02-22T21:07:10.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj4 testdata.516403 2026-02-22T21:07:10.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj5 testdata.516403 2026-02-22T21:07:10.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj6 testdata.516403 2026-02-22T21:07:10.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj7 testdata.516403 2026-02-22T21:07:10.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj8 testdata.516403 2026-02-22T21:07:10.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj9 testdata.516403 2026-02-22T21:07:10.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj10 testdata.516403 2026-02-22T21:07:10.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj11 testdata.516403 2026-02-22T21:07:10.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj12 testdata.516403 2026-02-22T21:07:10.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj13 testdata.516403 2026-02-22T21:07:10.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj14 testdata.516403 2026-02-22T21:07:10.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:10.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj15 testdata.516403 2026-02-22T21:07:11.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj16 testdata.516403 2026-02-22T21:07:11.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj17 testdata.516403 2026-02-22T21:07:11.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj18 testdata.516403 2026-02-22T21:07:11.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj19 testdata.516403 2026-02-22T21:07:11.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj20 testdata.516403 2026-02-22T21:07:11.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj21 testdata.516403 2026-02-22T21:07:11.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj22 testdata.516403 2026-02-22T21:07:11.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj23 testdata.516403 2026-02-22T21:07:11.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj24 testdata.516403 2026-02-22T21:07:11.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj25 testdata.516403 2026-02-22T21:07:11.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj26 testdata.516403 2026-02-22T21:07:11.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj27 testdata.516403 2026-02-22T21:07:11.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj28 testdata.516403 2026-02-22T21:07:11.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj29 testdata.516403 2026-02-22T21:07:11.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj30 testdata.516403 2026-02-22T21:07:11.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj31 testdata.516403 2026-02-22T21:07:11.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj32 testdata.516403 2026-02-22T21:07:11.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj33 testdata.516403 2026-02-22T21:07:11.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj34 testdata.516403 2026-02-22T21:07:11.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj35 testdata.516403 2026-02-22T21:07:11.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj36 testdata.516403 2026-02-22T21:07:11.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj37 testdata.516403 2026-02-22T21:07:11.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj38 testdata.516403 2026-02-22T21:07:11.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj39 testdata.516403 2026-02-22T21:07:11.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj40 testdata.516403 2026-02-22T21:07:11.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj41 testdata.516403 2026-02-22T21:07:11.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj42 testdata.516403 2026-02-22T21:07:11.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj43 testdata.516403 2026-02-22T21:07:11.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj44 testdata.516403 2026-02-22T21:07:11.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj45 testdata.516403 2026-02-22T21:07:11.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj46 testdata.516403 2026-02-22T21:07:11.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj47 testdata.516403 2026-02-22T21:07:11.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj48 testdata.516403 2026-02-22T21:07:11.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj49 testdata.516403 2026-02-22T21:07:11.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj50 testdata.516403 2026-02-22T21:07:11.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj51 testdata.516403 2026-02-22T21:07:11.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj52 testdata.516403 2026-02-22T21:07:11.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj53 testdata.516403 2026-02-22T21:07:11.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj54 testdata.516403 2026-02-22T21:07:11.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj55 testdata.516403 2026-02-22T21:07:11.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj56 testdata.516403 2026-02-22T21:07:11.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj57 testdata.516403 2026-02-22T21:07:11.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj58 testdata.516403 2026-02-22T21:07:11.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:11.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj59 testdata.516403 2026-02-22T21:07:12.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj60 testdata.516403 2026-02-22T21:07:12.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj61 testdata.516403 2026-02-22T21:07:12.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj62 testdata.516403 2026-02-22T21:07:12.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj63 testdata.516403 2026-02-22T21:07:12.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj64 testdata.516403 2026-02-22T21:07:12.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj65 testdata.516403 2026-02-22T21:07:12.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj66 testdata.516403 2026-02-22T21:07:12.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj67 testdata.516403 2026-02-22T21:07:12.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj68 testdata.516403 2026-02-22T21:07:12.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj69 testdata.516403 2026-02-22T21:07:12.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj70 testdata.516403 2026-02-22T21:07:12.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj71 testdata.516403 2026-02-22T21:07:12.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj72 testdata.516403 2026-02-22T21:07:12.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj73 testdata.516403 2026-02-22T21:07:12.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj74 testdata.516403 2026-02-22T21:07:12.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj75 testdata.516403 2026-02-22T21:07:12.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj76 testdata.516403 2026-02-22T21:07:12.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj77 testdata.516403 2026-02-22T21:07:12.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj78 testdata.516403 2026-02-22T21:07:12.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj79 testdata.516403 2026-02-22T21:07:12.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj80 testdata.516403 2026-02-22T21:07:12.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj81 testdata.516403 2026-02-22T21:07:12.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj82 testdata.516403 2026-02-22T21:07:12.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj83 testdata.516403 2026-02-22T21:07:12.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj84 testdata.516403 2026-02-22T21:07:12.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj85 testdata.516403 2026-02-22T21:07:12.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj86 testdata.516403 2026-02-22T21:07:12.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj87 testdata.516403 2026-02-22T21:07:12.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj88 testdata.516403 2026-02-22T21:07:12.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj89 testdata.516403 2026-02-22T21:07:12.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:477: TEST_just_deep_scrubs: for i in `seq 1 $objects` 2026-02-22T21:07:12.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:479: TEST_just_deep_scrubs: rados -p test put obj90 testdata.516403 2026-02-22T21:07:12.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:481: TEST_just_deep_scrubs: rm -f testdata.516403 2026-02-22T21:07:12.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:486: TEST_just_deep_scrubs: ceph tell 'osd.*' config set osd_scrub_retry_after_noscrub 2 2026-02-22T21:07:12.796 INFO:tasks.workunit.client.0.trial150.stdout:osd.0: { 2026-02-22T21:07:12.796 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_retry_after_noscrub = '' (not observed, change may require restart) " 2026-02-22T21:07:12.796 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:12.803 INFO:tasks.workunit.client.0.trial150.stdout:osd.1: { 2026-02-22T21:07:12.804 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_retry_after_noscrub = '' (not observed, change may require restart) " 2026-02-22T21:07:12.804 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:12.811 INFO:tasks.workunit.client.0.trial150.stdout:osd.2: { 2026-02-22T21:07:12.811 INFO:tasks.workunit.client.0.trial150.stdout: "success": "osd_scrub_retry_after_noscrub = '' (not observed, change may require restart) " 2026-02-22T21:07:12.811 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:12.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:487: TEST_just_deep_scrubs: ceph osd set noscrub 2026-02-22T21:07:13.026 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T21:07:13.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:488: TEST_just_deep_scrubs: ceph osd set nodeep-scrub 2026-02-22T21:07:13.231 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is set 2026-02-22T21:07:13.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:489: TEST_just_deep_scrubs: sleep 6 2026-02-22T21:07:19.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:490: TEST_just_deep_scrubs: date -Ins 2026-02-22T21:07:19.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:490: TEST_just_deep_scrubs: local now_is=2026-02-22T21:07:19,240885683+00:00 2026-02-22T21:07:19.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:491: TEST_just_deep_scrubs: declare -A sched_data 2026-02-22T21:07:19.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:492: TEST_just_deep_scrubs: local pgid=1.2 2026-02-22T21:07:19.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:495: TEST_just_deep_scrubs: set_query_debug 1.2 2026-02-22T21:07:19.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:417: set_query_debug: local pgid=1.2 2026-02-22T21:07:19.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: ceph pg dump pgs_brief 2026-02-22T21:07:19.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: awk -v 'pg=^1.2' -n -e '$0 ~ pg { print(gensub(/[^0-9]*([0-9]+).*/,"\\1","g",$5)); }' 2026-02-22T21:07:19.393 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs_brief 2026-02-22T21:07:19.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:419: set_query_debug: local prim_osd=0 2026-02-22T21:07:19.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:421: set_query_debug: echo 'Setting scrub debug data. Primary for 1.2 is 0' 2026-02-22T21:07:19.404 INFO:tasks.workunit.client.0.trial150.stdout:Setting scrub debug data. Primary for 1.2 is 0 2026-02-22T21:07:19.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: get_asok_path osd.0 2026-02-22T21:07:19.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-22T21:07:19.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-22T21:07:19.405 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:07:19.406 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:19.406 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:19.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-osd.0.asok 2026-02-22T21:07:19.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: CEPH_ARGS= 2026-02-22T21:07:19.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:422: set_query_debug: ceph --format=json daemon /tmp/ceph-asok.516403/ceph-osd.0.asok scrubdebug 1.2 set sessions 2026-02-22T21:07:19.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:497: TEST_just_deep_scrubs: extract_published_sch 1.2 2026-02-22T21:07:19,240885683+00:00 2026-02-22T21:07:19,240885683+00:00 sched_data 2026-02-22T21:07:19.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2026-02-22T21:07:19.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sched_data 2026-02-22T21:07:19.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2026-02-22T21:07:19,240885683+00:00 2026-02-22T21:07:19.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2026-02-22T21:07:19,240885683+00:00 2026-02-22T21:07:19.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2026-02-22T21:07:19.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2026-02-22T21:07:19.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2026-02-22T21:07:19.622 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:07:19.632 INFO:tasks.workunit.client.0.trial150.stdout:{"success":true}from pg dump pg: { 2026-02-22T21:07:19.633 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_pg_state": "active+clean", 2026-02-22T21:07:19.633 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_state_has_scrubbing": false, 2026-02-22T21:07:19.633 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_last_duration": 0, 2026-02-22T21:07:19.633 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-22T21:07:19.633 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule_at": "2026-02-23T21:07:04.615389+0000", 2026-02-22T21:07:19.633 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_is_future": true, 2026-02-22T21:07:19.633 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_vs_date": true, 2026-02-22T21:07:19.633 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_reported_epoch": 20, 2026-02-22T21:07:19.633 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_seq": 39 2026-02-22T21:07:19.633 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:19.634 INFO:tasks.workunit.client.0.trial150.stdout:query output: 2026-02-22T21:07:19.705 INFO:tasks.workunit.client.0.trial150.stdout: "scrubber": { 2026-02-22T21:07:19.705 INFO:tasks.workunit.client.0.trial150.stdout: "active": false, 2026-02-22T21:07:19.705 INFO:tasks.workunit.client.0.trial150.stdout: "must_scrub": false, 2026-02-22T21:07:19.705 INFO:tasks.workunit.client.0.trial150.stdout: "must_deep_scrub": false, 2026-02-22T21:07:19.705 INFO:tasks.workunit.client.0.trial150.stdout: "must_repair": false, 2026-02-22T21:07:19.705 INFO:tasks.workunit.client.0.trial150.stdout: "scrub_reg_stamp": "2026-02-23T21:07:04.615389+0000", 2026-02-22T21:07:19.705 INFO:tasks.workunit.client.0.trial150.stdout: "schedule": "scrub scheduled @ 2026-02-23T21:07:04.615 (2026-02-23T21:07:04.615)", 2026-02-22T21:07:19.705 INFO:tasks.workunit.client.0.trial150.stdout: "test_sequence": 540672 2026-02-22T21:07:19.705 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:19.706 INFO:tasks.workunit.client.0.trial150.stdout: "agent_state": {} 2026-02-22T21:07:19.787 INFO:tasks.workunit.client.0.trial150.stdout:( 2026-02-22T21:07:19.787 INFO:tasks.workunit.client.0.trial150.stdout:[query_epoch]=20 2026-02-22T21:07:19.787 INFO:tasks.workunit.client.0.trial150.stdout:[query_seq]=40 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[query_active]=false 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule]='scrub scheduled' 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule_at]='2026-02-23T21:07:04.615' 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[query_target_at]='2026-02-23T21:07:04.615' 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_duration]=0 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_stamp]='2026-02-22T21:07:04.615389+0000' 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_scrub]='0x0' 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[query_is_future]=true 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[query_vs_date]=true 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[query_scrub_seq]=540672 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_pg_state]='active+clean' 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_state_has_scrubbing]=false 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_last_duration]=0 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule_at]='2026-02-23T21:07:04.615389+0000' 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_is_future]=true 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_vs_date]=true 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_reported_epoch]=20 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_seq]=39 2026-02-22T21:07:19.788 INFO:tasks.workunit.client.0.trial150.stdout:) 2026-02-22T21:07:19.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:498: TEST_just_deep_scrubs: local saved_last_stamp=2026-02-22T21:07:04.615389+0000 2026-02-22T21:07:19.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:499: TEST_just_deep_scrubs: local dbg_counter_at_start=540672 2026-02-22T21:07:19.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:500: TEST_just_deep_scrubs: echo 'test counter @ start: 540672' 2026-02-22T21:07:19.801 INFO:tasks.workunit.client.0.trial150.stdout:test counter @ start: 540672 2026-02-22T21:07:19.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:502: TEST_just_deep_scrubs: ceph tell 1.2 schedule-deep-scrub 2026-02-22T21:07:19.861 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:07:19.861 INFO:tasks.workunit.client.0.trial150.stdout: "deep": true, 2026-02-22T21:07:19.861 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T21:07:19.861 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-11T16:19:09.856383+0000" 2026-02-22T21:07:19.861 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:19.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:504: TEST_just_deep_scrubs: sleep 5 2026-02-22T21:07:24.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:506: TEST_just_deep_scrubs: ceph pg dump pgs --format=json-pretty 2026-02-22T21:07:24.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:506: TEST_just_deep_scrubs: jq -r '.pg_stats[] | "\(.pgid) \(.stat_sum.num_objects)"' 2026-02-22T21:07:25.016 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:07:25.025 INFO:tasks.workunit.client.0.trial150.stdout:1.3 20 2026-02-22T21:07:25.025 INFO:tasks.workunit.client.0.trial150.stdout:1.2 20 2026-02-22T21:07:25.025 INFO:tasks.workunit.client.0.trial150.stdout:1.1 25 2026-02-22T21:07:25.025 INFO:tasks.workunit.client.0.trial150.stdout:1.0 25 2026-02-22T21:07:25.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:507: TEST_just_deep_scrubs: ceph pg 1.2 query --format=json-pretty 2026-02-22T21:07:25.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:507: TEST_just_deep_scrubs: jq -r .info.stats.stat_sum.num_objects 2026-02-22T21:07:25.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:507: TEST_just_deep_scrubs: echo 'Objects # in pg 1.2: ' 20 2026-02-22T21:07:25.099 INFO:tasks.workunit.client.0.trial150.stdout:Objects # in pg 1.2: 20 2026-02-22T21:07:25.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:509: TEST_just_deep_scrubs: declare -A sc_data_2 2026-02-22T21:07:25.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:510: TEST_just_deep_scrubs: extract_published_sch 1.2 2026-02-22T21:07:19,240885683+00:00 2026-02-22T21:07:19,240885683+00:00 sc_data_2 2026-02-22T21:07:25.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2026-02-22T21:07:25.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sc_data_2 2026-02-22T21:07:25.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2026-02-22T21:07:19,240885683+00:00 2026-02-22T21:07:25.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2026-02-22T21:07:19,240885683+00:00 2026-02-22T21:07:25.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2026-02-22T21:07:25.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2026-02-22T21:07:25.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2026-02-22T21:07:25.245 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:07:25.255 INFO:tasks.workunit.client.0.trial150.stdout:from pg dump pg: { 2026-02-22T21:07:25.255 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_pg_state": "active+clean", 2026-02-22T21:07:25.255 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_state_has_scrubbing": false, 2026-02-22T21:07:25.255 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_last_duration": 0, 2026-02-22T21:07:25.255 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule": "periodic deep scrub scheduled", 2026-02-22T21:07:25.255 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule_at": "2026-02-22T21:07:22.484904+0000", 2026-02-22T21:07:25.255 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_is_future": true, 2026-02-22T21:07:25.255 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_vs_date": true, 2026-02-22T21:07:25.255 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_reported_epoch": 20, 2026-02-22T21:07:25.255 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_seq": 44 2026-02-22T21:07:25.255 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:25.255 INFO:tasks.workunit.client.0.trial150.stdout:query output: 2026-02-22T21:07:25.330 INFO:tasks.workunit.client.0.trial150.stdout: "scrubber": { 2026-02-22T21:07:25.330 INFO:tasks.workunit.client.0.trial150.stdout: "active": false, 2026-02-22T21:07:25.331 INFO:tasks.workunit.client.0.trial150.stdout: "must_scrub": false, 2026-02-22T21:07:25.331 INFO:tasks.workunit.client.0.trial150.stdout: "must_deep_scrub": false, 2026-02-22T21:07:25.331 INFO:tasks.workunit.client.0.trial150.stdout: "must_repair": false, 2026-02-22T21:07:25.331 INFO:tasks.workunit.client.0.trial150.stdout: "scrub_reg_stamp": "2026-02-22T21:07:26.509269+0000", 2026-02-22T21:07:25.331 INFO:tasks.workunit.client.0.trial150.stdout: "schedule": "deep scrub scheduled @ 2026-02-22T21:07:26.509 (2026-02-19T08:06:30.195)", 2026-02-22T21:07:25.331 INFO:tasks.workunit.client.0.trial150.stdout: "test_sequence": 540672 2026-02-22T21:07:25.331 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:25.331 INFO:tasks.workunit.client.0.trial150.stdout: "agent_state": {} 2026-02-22T21:07:25.411 INFO:tasks.workunit.client.0.trial150.stdout:( 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_epoch]=20 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_seq]=45 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_active]=false 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule]='deep scrub scheduled' 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule_at]='2026-02-22T21:07:26.509' 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_target_at]='2026-02-19T08:06:30.195' 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_duration]=0 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_stamp]='2026-02-22T21:07:04.615389+0000' 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_scrub]='0x0' 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_is_future]=false 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_vs_date]=true 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[query_scrub_seq]=540672 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_pg_state]='active+clean' 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_state_has_scrubbing]=false 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_last_duration]=0 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule]='periodic deep scrub scheduled' 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule_at]='2026-02-22T21:07:22.484904+0000' 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_is_future]=true 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_vs_date]=true 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_reported_epoch]=20 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_seq]=44 2026-02-22T21:07:25.412 INFO:tasks.workunit.client.0.trial150.stdout:) 2026-02-22T21:07:25.424 INFO:tasks.workunit.client.0.trial150.stdout:test counter @ should show no change: 540672 2026-02-22T21:07:25.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:511: TEST_just_deep_scrubs: echo 'test counter @ should show no change: ' 540672 2026-02-22T21:07:25.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:512: TEST_just_deep_scrubs: (( 0 == 0 )) 2026-02-22T21:07:25.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:513: TEST_just_deep_scrubs: (( 540672 == 540672 )) 2026-02-22T21:07:25.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:516: TEST_just_deep_scrubs: ceph osd unset nodeep-scrub 2026-02-22T21:07:25.628 INFO:tasks.workunit.client.0.trial150.stderr:nodeep-scrub is unset 2026-02-22T21:07:25.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:517: TEST_just_deep_scrubs: sleep 8 2026-02-22T21:07:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2026-02-22T21:07:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:518: TEST_just_deep_scrubs: declare -A expct_qry_duration 2026-02-22T21:07:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:519: TEST_just_deep_scrubs: sc_data_2=() 2026-02-22T21:07:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:520: TEST_just_deep_scrubs: extract_published_sch 1.2 2026-02-22T21:07:19,240885683+00:00 2026-02-22T21:07:19,240885683+00:00 sc_data_2 2026-02-22T21:07:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:17: extract_published_sch: local pgn=1.2 2026-02-22T21:07:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:18: extract_published_sch: local -n dict=sc_data_2 2026-02-22T21:07:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:19: extract_published_sch: local current_time=2026-02-22T21:07:19,240885683+00:00 2026-02-22T21:07:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:20: extract_published_sch: local extra_time=2026-02-22T21:07:19,240885683+00:00 2026-02-22T21:07:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:21: extract_published_sch: local extr_dbg=2 2026-02-22T21:07:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:24: extract_published_sch: local saved_echo_flag=x 2026-02-22T21:07:33.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:25: extract_published_sch: set +x 2026-02-22T21:07:33.790 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:07:33.800 INFO:tasks.workunit.client.0.trial150.stdout:from pg dump pg: { 2026-02-22T21:07:33.800 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_pg_state": "active+clean", 2026-02-22T21:07:33.800 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_state_has_scrubbing": false, 2026-02-22T21:07:33.800 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_last_duration": 1, 2026-02-22T21:07:33.800 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-22T21:07:33.800 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule_at": "2026-02-23T21:07:27.481804+0000", 2026-02-22T21:07:33.800 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_is_future": true, 2026-02-22T21:07:33.800 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_vs_date": true, 2026-02-22T21:07:33.800 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_reported_epoch": 21, 2026-02-22T21:07:33.800 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_seq": 57 2026-02-22T21:07:33.800 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:33.800 INFO:tasks.workunit.client.0.trial150.stdout:query output: 2026-02-22T21:07:33.873 INFO:tasks.workunit.client.0.trial150.stdout: "scrubber": { 2026-02-22T21:07:33.873 INFO:tasks.workunit.client.0.trial150.stdout: "active": false, 2026-02-22T21:07:33.873 INFO:tasks.workunit.client.0.trial150.stdout: "must_scrub": false, 2026-02-22T21:07:33.873 INFO:tasks.workunit.client.0.trial150.stdout: "must_deep_scrub": false, 2026-02-22T21:07:33.874 INFO:tasks.workunit.client.0.trial150.stdout: "must_repair": false, 2026-02-22T21:07:33.874 INFO:tasks.workunit.client.0.trial150.stdout: "scrub_reg_stamp": "2026-02-23T21:07:27.481804+0000", 2026-02-22T21:07:33.874 INFO:tasks.workunit.client.0.trial150.stdout: "schedule": "scrub scheduled @ 2026-02-23T21:07:27.481 (2026-02-23T21:07:27.481)", 2026-02-22T21:07:33.874 INFO:tasks.workunit.client.0.trial150.stdout: "test_sequence": 540675 2026-02-22T21:07:33.874 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:33.874 INFO:tasks.workunit.client.0.trial150.stdout: "agent_state": {} 2026-02-22T21:07:33.957 INFO:tasks.workunit.client.0.trial150.stdout:( 2026-02-22T21:07:33.957 INFO:tasks.workunit.client.0.trial150.stdout:[query_epoch]=21 2026-02-22T21:07:33.958 INFO:tasks.workunit.client.0.trial150.stdout:[query_seq]=58 2026-02-22T21:07:33.958 INFO:tasks.workunit.client.0.trial150.stdout:[query_active]=false 2026-02-22T21:07:33.958 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule]='scrub scheduled' 2026-02-22T21:07:33.958 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule_at]='2026-02-23T21:07:27.481' 2026-02-22T21:07:33.958 INFO:tasks.workunit.client.0.trial150.stdout:[query_target_at]='2026-02-23T21:07:27.481' 2026-02-22T21:07:33.958 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_duration]=1 2026-02-22T21:07:33.958 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_stamp]='2026-02-22T21:07:27.481804+0000' 2026-02-22T21:07:33.958 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_scrub]='18x20' 2026-02-22T21:07:33.958 INFO:tasks.workunit.client.0.trial150.stdout:[query_is_future]=true 2026-02-22T21:07:33.958 INFO:tasks.workunit.client.0.trial150.stdout:[query_vs_date]=true 2026-02-22T21:07:33.958 INFO:tasks.workunit.client.0.trial150.stdout:[query_scrub_seq]=540675 2026-02-22T21:07:33.959 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_pg_state]='active+clean' 2026-02-22T21:07:33.959 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_state_has_scrubbing]=false 2026-02-22T21:07:33.959 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_last_duration]=1 2026-02-22T21:07:33.959 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-22T21:07:33.959 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule_at]='2026-02-23T21:07:27.481804+0000' 2026-02-22T21:07:33.959 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_is_future]=true 2026-02-22T21:07:33.959 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_vs_date]=true 2026-02-22T21:07:33.959 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_reported_epoch]=21 2026-02-22T21:07:33.959 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_seq]=57 2026-02-22T21:07:33.959 INFO:tasks.workunit.client.0.trial150.stdout:) 2026-02-22T21:07:33.968 INFO:tasks.workunit.client.0.trial150.stdout:test counter @ should be higher than before the unset: 540675 2026-02-22T21:07:33.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:521: TEST_just_deep_scrubs: echo 'test counter @ should be higher than before the unset: ' 540675 2026-02-22T21:07:33.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:523: TEST_just_deep_scrubs: expct_qry_duration=(['query_last_duration']='0' ['query_last_duration_neg']='not0') 2026-02-22T21:07:33.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:523: TEST_just_deep_scrubs: declare -A expct_qry_duration 2026-02-22T21:07:33.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:524: TEST_just_deep_scrubs: sc_data_2=() 2026-02-22T21:07:33.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:525: TEST_just_deep_scrubs: wait_any_cond 1.2 10 2026-02-22T21:07:04.615389+0000 expct_qry_duration 'WaitingAfterScrub ' sc_data_2 2026-02-22T21:07:33.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:106: wait_any_cond: local pgid=1.2 2026-02-22T21:07:33.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:107: wait_any_cond: local retries=10 2026-02-22T21:07:33.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:108: wait_any_cond: local cmp_date=2026-02-22T21:07:04.615389+0000 2026-02-22T21:07:33.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:109: wait_any_cond: local -n ep=expct_qry_duration 2026-02-22T21:07:33.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:110: wait_any_cond: local -n out_array=sc_data_2 2026-02-22T21:07:33.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:111: wait_any_cond: local -A sc_data 2026-02-22T21:07:33.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:112: wait_any_cond: local extr_dbg=2 2026-02-22T21:07:33.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:115: wait_any_cond: local saved_echo_flag=x 2026-02-22T21:07:33.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:116: wait_any_cond: set +x 2026-02-22T21:07:33.971 INFO:tasks.workunit.client.0.trial150.stdout:waiting for any condition (WaitingAfterScrub ): pg:1.2 dt:2026-02-22T21:07:04.615389+0000 (10 retries) 2026-02-22T21:07:34.616 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:07:34.626 INFO:tasks.workunit.client.0.trial150.stdout:from pg dump pg: { 2026-02-22T21:07:34.626 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_pg_state": "active+clean", 2026-02-22T21:07:34.626 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_state_has_scrubbing": false, 2026-02-22T21:07:34.626 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_last_duration": 1, 2026-02-22T21:07:34.626 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule": "periodic scrub scheduled", 2026-02-22T21:07:34.627 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_schedule_at": "2026-02-23T21:07:27.481804+0000", 2026-02-22T21:07:34.627 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_is_future": true, 2026-02-22T21:07:34.627 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_vs_date": true, 2026-02-22T21:07:34.627 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_reported_epoch": 21, 2026-02-22T21:07:34.627 INFO:tasks.workunit.client.0.trial150.stdout: "dmp_seq": 57 2026-02-22T21:07:34.627 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:34.627 INFO:tasks.workunit.client.0.trial150.stdout:query output: 2026-02-22T21:07:34.699 INFO:tasks.workunit.client.0.trial150.stdout: "scrubber": { 2026-02-22T21:07:34.699 INFO:tasks.workunit.client.0.trial150.stdout: "active": false, 2026-02-22T21:07:34.699 INFO:tasks.workunit.client.0.trial150.stdout: "must_scrub": false, 2026-02-22T21:07:34.699 INFO:tasks.workunit.client.0.trial150.stdout: "must_deep_scrub": false, 2026-02-22T21:07:34.699 INFO:tasks.workunit.client.0.trial150.stdout: "must_repair": false, 2026-02-22T21:07:34.699 INFO:tasks.workunit.client.0.trial150.stdout: "scrub_reg_stamp": "2026-02-23T21:07:27.481804+0000", 2026-02-22T21:07:34.699 INFO:tasks.workunit.client.0.trial150.stdout: "schedule": "scrub scheduled @ 2026-02-23T21:07:27.481 (2026-02-23T21:07:27.481)", 2026-02-22T21:07:34.699 INFO:tasks.workunit.client.0.trial150.stdout: "test_sequence": 540675 2026-02-22T21:07:34.699 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:34.699 INFO:tasks.workunit.client.0.trial150.stdout: "agent_state": {} 2026-02-22T21:07:34.783 INFO:tasks.workunit.client.0.trial150.stdout:( 2026-02-22T21:07:34.783 INFO:tasks.workunit.client.0.trial150.stdout:[query_epoch]=21 2026-02-22T21:07:34.783 INFO:tasks.workunit.client.0.trial150.stdout:[query_seq]=58 2026-02-22T21:07:34.784 INFO:tasks.workunit.client.0.trial150.stdout:[query_active]=false 2026-02-22T21:07:34.784 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule]='scrub scheduled' 2026-02-22T21:07:34.784 INFO:tasks.workunit.client.0.trial150.stdout:[query_schedule_at]='2026-02-23T21:07:27.481' 2026-02-22T21:07:34.784 INFO:tasks.workunit.client.0.trial150.stdout:[query_target_at]='2026-02-23T21:07:27.481' 2026-02-22T21:07:34.784 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_duration]=1 2026-02-22T21:07:34.784 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_stamp]='2026-02-22T21:07:27.481804+0000' 2026-02-22T21:07:34.784 INFO:tasks.workunit.client.0.trial150.stdout:[query_last_scrub]='18x20' 2026-02-22T21:07:34.784 INFO:tasks.workunit.client.0.trial150.stdout:[query_is_future]=true 2026-02-22T21:07:34.784 INFO:tasks.workunit.client.0.trial150.stdout:[query_vs_date]=true 2026-02-22T21:07:34.784 INFO:tasks.workunit.client.0.trial150.stdout:[query_scrub_seq]=540675 2026-02-22T21:07:34.784 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_pg_state]='active+clean' 2026-02-22T21:07:34.785 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_state_has_scrubbing]=false 2026-02-22T21:07:34.785 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_last_duration]=1 2026-02-22T21:07:34.785 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule]='periodic scrub scheduled' 2026-02-22T21:07:34.785 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_schedule_at]='2026-02-23T21:07:27.481804+0000' 2026-02-22T21:07:34.785 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_is_future]=true 2026-02-22T21:07:34.785 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_vs_date]=true 2026-02-22T21:07:34.785 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_reported_epoch]=21 2026-02-22T21:07:34.785 INFO:tasks.workunit.client.0.trial150.stdout:[dmp_seq]=57 2026-02-22T21:07:34.785 INFO:tasks.workunit.client.0.trial150.stdout:) 2026-02-22T21:07:34.794 INFO:tasks.workunit.client.0.trial150.stdout:--> loop: 1 ~ false / 58 / 57 / true / 2026-02-22T21:07:27.481804+0000 / scrub scheduled %%% query_last_duration_neg query_last_duration 2026-02-22T21:07:34.794 INFO:tasks.workunit.client.0.trial150.stdout:key is query_last_duration: negation:1 # expected: 0 # in actual: 1 2026-02-22T21:07:34.794 INFO:tasks.workunit.client.0.trial150.stdout:WaitingAfterScrub - 'query_last_duration' actual value (1) matches expected (0) (negation: 1) 2026-02-22T21:07:34.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/scrub-helpers.sh:157: wait_any_cond: return 0 2026-02-22T21:07:34.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:526: TEST_just_deep_scrubs: perf_counters td/osd-scrub-test 3 2026-02-22T21:07:34.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-22T21:07:34.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-22T21:07:34.795 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-22T21:07:34.796 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-22T21:07:34.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:07:34.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-22T21:07:34.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:07:34.869 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:07:34.869 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:07:34.869 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:34.869 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:34.869 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:07:34.869 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:07:34.869 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:34.869 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:34.869 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:34.870 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:34.870 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:34.870 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:34.870 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:34.870 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.870 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.870 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:34.870 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:07:34.870 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:34.870 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:34.870 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:07:34.870 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:07:34.871 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:34.871 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:34.871 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:34.871 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 2, 2026-02-22T21:07:34.871 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:34.871 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:34.871 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:34.871 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.871 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.871 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:34.871 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:07:34.871 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:34.872 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:34.872 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:07:34.872 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:07:34.872 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:34.872 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:34.872 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:34.872 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:34.873 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:34.873 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:34.873 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:34.873 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.873 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.873 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:34.874 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:07:34.874 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:34.874 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:34.874 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:07:34.874 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:07:34.874 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:34.875 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:34.875 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:34.875 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:34.875 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:34.875 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:34.875 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:34.876 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.876 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.876 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:07:34.876 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:34.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:07:34.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-22T21:07:34.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:07:34.940 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:07:34.940 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:07:34.940 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:34.940 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:34.940 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:07:34.940 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:07:34.940 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:34.940 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:07:34.941 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:34.942 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:34.943 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:07:34.944 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:34.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:07:34.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-22T21:07:34.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:07:35.013 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:07:35.014 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:35.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_just_deep_scrubs ------------------' 2026-02-22T21:07:35.016 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Teardown Test TEST_just_deep_scrubs ------------------ 2026-02-22T21:07:35.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-22T21:07:35.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:07:35.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:07:35.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:07:35.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:07:35.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:07:35.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:07:35.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:07:35.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:07:35.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:07:35.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:07:35.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:07:35.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:07:35.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:07:35.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:07:35.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:07:35.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:07:35.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:07:35.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:07:35.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:07:35.132 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:07:35.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:07:35.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:07:35.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:07:35.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:35.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:35.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:07:35.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:07:35.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:07:35.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_just_deep_scrubs ------------------' 2026-02-22T21:07:35.148 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Complete Test TEST_just_deep_scrubs ------------------ 2026-02-22T21:07:35.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-22T21:07:35.149 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Prepare Test TEST_pg_dump_objects_scrubbed ------------------- 2026-02-22T21:07:35.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_pg_dump_objects_scrubbed -------------------' 2026-02-22T21:07:35.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-22T21:07:35.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-22T21:07:35.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-22T21:07:35.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:07:35.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:07:35.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:07:35.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:07:35.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:07:35.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:07:35.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:07:35.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:07:35.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:07:35.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:07:35.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:07:35.153 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:07:35.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:07:35.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:07:35.154 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:07:35.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:07:35.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:07:35.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:07:35.156 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:07:35.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:07:35.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:07:35.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:07:35.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:07:35.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:35.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:35.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:07:35.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:07:35.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:07:35.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-22T21:07:35.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T21:07:35.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:35.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:35.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.516403 2026-02-22T21:07:35.164 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T21:07:35.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:07:35.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T21:07:35.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-22T21:07:35.165 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Run Test TEST_pg_dump_objects_scrubbed ----------------------- 2026-02-22T21:07:35.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_pg_dump_objects_scrubbed -----------------------' 2026-02-22T21:07:35.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_pg_dump_objects_scrubbed td/osd-scrub-test 2026-02-22T21:07:35.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:653: TEST_pg_dump_objects_scrubbed: local dir=td/osd-scrub-test 2026-02-22T21:07:35.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:654: TEST_pg_dump_objects_scrubbed: local poolname=test 2026-02-22T21:07:35.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:655: TEST_pg_dump_objects_scrubbed: local OSDS=3 2026-02-22T21:07:35.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:656: TEST_pg_dump_objects_scrubbed: local objects=15 2026-02-22T21:07:35.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:657: TEST_pg_dump_objects_scrubbed: local timeout=10 2026-02-22T21:07:35.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:659: TEST_pg_dump_objects_scrubbed: TESTDATA=testdata.516403 2026-02-22T21:07:35.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:661: TEST_pg_dump_objects_scrubbed: setup td/osd-scrub-test 2026-02-22T21:07:35.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-22T21:07:35.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-22T21:07:35.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:07:35.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:07:35.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:07:35.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:07:35.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:07:35.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:07:35.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:07:35.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:07:35.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:07:35.169 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:07:35.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:07:35.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:07:35.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:07:35.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:07:35.171 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:07:35.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:07:35.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:07:35.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:07:35.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:07:35.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:07:35.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:07:35.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:07:35.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:07:35.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:35.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:35.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:07:35.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:07:35.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:07:35.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-22T21:07:35.180 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T21:07:35.180 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:35.180 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:35.180 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.516403 2026-02-22T21:07:35.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T21:07:35.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:07:35.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T21:07:35.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-22T21:07:35.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:662: TEST_pg_dump_objects_scrubbed: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-22T21:07:35.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-22T21:07:35.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T21:07:35.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T21:07:35.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T21:07:35.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-22T21:07:35.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-22T21:07:35.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T21:07:35.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:07:35.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:07:35.205 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:07:35.205 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:35.205 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:35.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:35.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-22T21:07:35.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T21:07:35.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T21:07:35.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:07:35.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:07:35.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T21:07:35.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T21:07:35.235 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:07:35.235 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:07:35.235 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:07:35.236 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:07:35.236 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:35.236 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:35.236 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T21:07:35.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:07:35.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get fsid 2026-02-22T21:07:35.289 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T21:07:35.289 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:07:35.289 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:07:35.289 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T21:07:35.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T21:07:35.290 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:07:35.290 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:07:35.290 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:07:35.290 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:07:35.290 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:35.290 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:35.291 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T21:07:35.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:07:35.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get mon_host 2026-02-22T21:07:35.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:663: TEST_pg_dump_objects_scrubbed: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-22T21:07:35.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-22T21:07:35.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T21:07:35.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T21:07:35.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T21:07:35.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-22T21:07:35.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:07:35.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T21:07:35.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:07:35.436 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:07:35.437 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:07:35.437 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:35.437 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:35.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:35.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:07:35.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-22T21:07:35.458 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: expr 3 - 1 2026-02-22T21:07:35.459 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: seq 0 2 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:666: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 0 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-22T21:07:35.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:07:35.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:07:35.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:07:35.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:07:35.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:07:35.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T21:07:35.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-22T21:07:35.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:07:35.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9974b8f8-944e-41ab-a179-af6f42dd0b1e 2026-02-22T21:07:35.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 9974b8f8-944e-41ab-a179-af6f42dd0b1e' 2026-02-22T21:07:35.464 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 9974b8f8-944e-41ab-a179-af6f42dd0b1e 2026-02-22T21:07:35.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:07:35.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAXcJtptAALHBAAzYbDNAEfaJJk4ZcMJdjoSg== 2026-02-22T21:07:35.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAXcJtptAALHBAAzYbDNAEfaJJk4ZcMJdjoSg=="}' 2026-02-22T21:07:35.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9974b8f8-944e-41ab-a179-af6f42dd0b1e -i td/osd-scrub-test/0/new.json 2026-02-22T21:07:35.565 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:07:35.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-22T21:07:35.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAXcJtptAALHBAAzYbDNAEfaJJk4ZcMJdjoSg== --osd-uuid 9974b8f8-944e-41ab-a179-af6f42dd0b1e 2026-02-22T21:07:35.590 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:35.583+0000 7fb19922fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:35.591 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:35.583+0000 7fb19922fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:35.592 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:35.587+0000 7fb19922fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:35.593 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:35.587+0000 7fb19922fa40 -1 bdev(0x558efe33cc00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:07:35.593 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:35.587+0000 7fb19922fa40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-22T21:07:36.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-22T21:07:36.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:07:36.361 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T21:07:36.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:07:36.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:07:36.567 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T21:07:36.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T21:07:36.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:07:36.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:07:36.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:07:36.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:07:36.582 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:36.575+0000 7fc9fbba6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:36.583 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:36.575+0000 7fc9fbba6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:36.584 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:36.575+0000 7fc9fbba6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:36.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T21:07:36.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:07:36.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T21:07:36.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:07:36.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:07:36.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:07:36.729 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:07:36.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:07:36.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:07:36.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:07:36.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:07:37.022 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:37.015+0000 7fc9fbba6a40 -1 Falling back to public interface 2026-02-22T21:07:37.587 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:37.579+0000 7fc9fbba6a40 -1 osd.0 0 log_to_monitors true 2026-02-22T21:07:37.883 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:07:37.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:07:37.884 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:07:37.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:07:37.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:07:37.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:07:38.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:07:39.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:07:39.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:07:39.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:07:39.047 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:07:39.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:07:39.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:07:39.208 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/179038581,v1:127.0.0.1:6803/179038581] [v2:127.0.0.1:6804/179038581,v1:127.0.0.1:6805/179038581] exists,up 9974b8f8-944e-41ab-a179-af6f42dd0b1e 2026-02-22T21:07:39.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:07:39.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:07:39.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:07:39.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:07:39.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:666: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 1 2026-02-22T21:07:39.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:07:39.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:07:39.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T21:07:39.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:07:39.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-22T21:07:39.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:39.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:39.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:07:39.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:07:39.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:07:39.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:07:39.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:07:39.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:07:39.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:07:39.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:07:39.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:07:39.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:07:39.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T21:07:39.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-22T21:07:39.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:07:39.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2065d207-8d69-477a-96a9-b783dfc597dc 2026-02-22T21:07:39.214 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 2065d207-8d69-477a-96a9-b783dfc597dc 2026-02-22T21:07:39.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2065d207-8d69-477a-96a9-b783dfc597dc' 2026-02-22T21:07:39.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:07:39.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAbcJtp9CASDRAAvBVX8R9/xBDvOD51YpbUdA== 2026-02-22T21:07:39.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAbcJtp9CASDRAAvBVX8R9/xBDvOD51YpbUdA=="}' 2026-02-22T21:07:39.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2065d207-8d69-477a-96a9-b783dfc597dc -i td/osd-scrub-test/1/new.json 2026-02-22T21:07:39.381 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:07:39.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-22T21:07:39.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbcJtp9CASDRAAvBVX8R9/xBDvOD51YpbUdA== --osd-uuid 2065d207-8d69-477a-96a9-b783dfc597dc 2026-02-22T21:07:39.418 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:39.411+0000 7feb6d66fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:39.419 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:39.411+0000 7feb6d66fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:39.420 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:39.411+0000 7feb6d66fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:39.420 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:39.415+0000 7feb6d66fa40 -1 bdev(0x556d1961cc00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:07:39.421 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:39.415+0000 7feb6d66fa40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-22T21:07:40.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-22T21:07:40.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:07:40.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:07:40.121 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T21:07:40.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:07:40.322 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T21:07:40.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T21:07:40.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:07:40.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:07:40.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:07:40.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:07:40.344 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:40.335+0000 7fb71fec9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:40.346 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:40.339+0000 7fb71fec9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:40.347 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:40.339+0000 7fb71fec9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:40.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T21:07:40.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:07:40.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T21:07:40.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:07:40.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:07:40.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:07:40.482 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:07:40.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:07:40.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:07:40.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:07:40.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:07:40.694 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:40.687+0000 7fb71fec9a40 -1 Falling back to public interface 2026-02-22T21:07:41.278 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:41.271+0000 7fb71fec9a40 -1 osd.1 0 log_to_monitors true 2026-02-22T21:07:41.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:07:41.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:07:41.640 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:07:41.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:07:41.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:07:41.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:07:41.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:07:42.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:07:42.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:07:42.800 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:07:42.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:07:42.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:07:42.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:07:42.958 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/700327598,v1:127.0.0.1:6811/700327598] [v2:127.0.0.1:6812/700327598,v1:127.0.0.1:6813/700327598] exists,up 2065d207-8d69-477a-96a9-b783dfc597dc 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:664: TEST_pg_dump_objects_scrubbed: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:666: TEST_pg_dump_objects_scrubbed: run_osd td/osd-scrub-test 2 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:07:42.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:07:42.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:07:42.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:07:42.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:07:42.960 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:07:42.960 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:42.960 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:42.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T21:07:42.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-22T21:07:42.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:07:42.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=653775f6-3b7f-4fc6-8707-4e17cffc1b63 2026-02-22T21:07:42.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 653775f6-3b7f-4fc6-8707-4e17cffc1b63' 2026-02-22T21:07:42.963 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 653775f6-3b7f-4fc6-8707-4e17cffc1b63 2026-02-22T21:07:42.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:07:42.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAecJtpUQ6/ORAAxzyKQMO0WU/gu1cwA0ocOw== 2026-02-22T21:07:42.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAecJtpUQ6/ORAAxzyKQMO0WU/gu1cwA0ocOw=="}' 2026-02-22T21:07:42.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 653775f6-3b7f-4fc6-8707-4e17cffc1b63 -i td/osd-scrub-test/2/new.json 2026-02-22T21:07:43.137 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:07:43.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-22T21:07:43.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAecJtpUQ6/ORAAxzyKQMO0WU/gu1cwA0ocOw== --osd-uuid 653775f6-3b7f-4fc6-8707-4e17cffc1b63 2026-02-22T21:07:43.174 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:43.167+0000 7f16d8936a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:43.175 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:43.167+0000 7f16d8936a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:43.176 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:43.167+0000 7f16d8936a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:43.176 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:43.171+0000 7f16d8936a40 -1 bdev(0x565374244c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:07:43.177 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:43.171+0000 7f16d8936a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-22T21:07:43.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-22T21:07:43.816 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:07:43.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:07:43.817 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T21:07:43.818 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:07:44.020 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T21:07:44.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T21:07:44.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:07:44.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:07:44.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:07:44.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:07:44.043 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:44.035+0000 7fbbe7c30a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:44.044 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:44.035+0000 7fbbe7c30a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:44.045 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:44.039+0000 7fbbe7c30a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:44.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T21:07:44.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:07:44.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T21:07:44.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:07:44.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:07:44.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:07:44.184 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:07:44.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:07:44.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:07:44.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:07:44.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:07:44.402 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:44.395+0000 7fbbe7c30a40 -1 Falling back to public interface 2026-02-22T21:07:44.994 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:44.987+0000 7fbbe7c30a40 -1 osd.2 0 log_to_monitors true 2026-02-22T21:07:45.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:07:45.347 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:07:45.348 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:07:45.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:07:45.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:07:45.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:07:45.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:07:46.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:07:46.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:07:46.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:07:46.507 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:07:46.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:07:46.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:07:46.658 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/420887466,v1:127.0.0.1:6819/420887466] [v2:127.0.0.1:6820/420887466,v1:127.0.0.1:6821/420887466] exists,up 653775f6-3b7f-4fc6-8707-4e17cffc1b63 2026-02-22T21:07:46.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:07:46.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:07:46.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:07:46.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:670: TEST_pg_dump_objects_scrubbed: create_pool test 1 1 2026-02-22T21:07:46.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-22T21:07:46.865 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T21:07:46.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T21:07:47.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:671: TEST_pg_dump_objects_scrubbed: wait_for_clean 2026-02-22T21:07:47.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:07:47.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:07:47.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:07:47.882 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:07:47.882 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:07:47.882 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:07:47.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:07:47.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:07:47.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:07:47.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:07:47.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:07:47.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:07:47.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:07:47.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:07:47.983 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:07:48.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:07:48.146 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:07:48.146 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:07:48.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:07:48.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:07:48.147 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:07:48.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-22T21:07:48.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-22T21:07:48.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T21:07:48.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:07:48.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:07:48.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705671 2026-02-22T21:07:48.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705671 2026-02-22T21:07:48.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705671' 2026-02-22T21:07:48.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:07:48.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:07:48.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542148 2026-02-22T21:07:48.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542148 2026-02-22T21:07:48.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-38654705671 2-60129542148' 2026-02-22T21:07:48.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:07:48.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-22T21:07:48.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:07:48.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:07:48.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-22T21:07:48.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:07:48.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-22T21:07:48.374 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836491 2026-02-22T21:07:48.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T21:07:48.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:07:48.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T21:07:48.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:07:49.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:07:49.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:07:49.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T21:07:49.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:07:50.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-22T21:07:50.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:07:50.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836491 2026-02-22T21:07:50.851 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:07:50.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705671 2026-02-22T21:07:50.851 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:07:50.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:07:50.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705671 2026-02-22T21:07:50.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:07:50.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705671 2026-02-22T21:07:50.855 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705671 2026-02-22T21:07:50.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705671' 2026-02-22T21:07:50.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:07:51.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705673 -lt 38654705671 2026-02-22T21:07:51.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:07:51.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542148 2026-02-22T21:07:51.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:07:51.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:07:51.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542148 2026-02-22T21:07:51.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:07:51.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542148 2026-02-22T21:07:51.032 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542148 2026-02-22T21:07:51.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2026-02-22T21:07:51.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:07:51.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542149 -lt 60129542148 2026-02-22T21:07:51.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:07:51.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:07:51.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:07:51.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:07:51.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:07:51.389 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:07:51.389 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:07:51.389 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:07:51.389 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:07:51.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:07:51.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:07:51.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:07:51.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:07:51.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:07:51.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:07:51.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:07:51.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:07:51.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:07:51.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: ceph osd dump 2026-02-22T21:07:51.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: grep '^pool.*['\'']test['\'']' 2026-02-22T21:07:51.745 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: awk '{ print $2 }' 2026-02-22T21:07:51.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:672: TEST_pg_dump_objects_scrubbed: poolid=1 2026-02-22T21:07:51.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:674: TEST_pg_dump_objects_scrubbed: dd if=/dev/urandom of=testdata.516403 bs=1032 count=1 2026-02-22T21:07:51.904 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T21:07:51.904 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T21:07:51.904 INFO:tasks.workunit.client.0.trial150.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.2611e-05 s, 31.6 MB/s 2026-02-22T21:07:51.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: seq 1 15 2026-02-22T21:07:51.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:51.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj1 testdata.516403 2026-02-22T21:07:51.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:51.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj2 testdata.516403 2026-02-22T21:07:51.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:51.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj3 testdata.516403 2026-02-22T21:07:51.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:51.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj4 testdata.516403 2026-02-22T21:07:51.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:51.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj5 testdata.516403 2026-02-22T21:07:52.016 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:52.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj6 testdata.516403 2026-02-22T21:07:52.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:52.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj7 testdata.516403 2026-02-22T21:07:52.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:52.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj8 testdata.516403 2026-02-22T21:07:52.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:52.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj9 testdata.516403 2026-02-22T21:07:52.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:52.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj10 testdata.516403 2026-02-22T21:07:52.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:52.138 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj11 testdata.516403 2026-02-22T21:07:52.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:52.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj12 testdata.516403 2026-02-22T21:07:52.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:52.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj13 testdata.516403 2026-02-22T21:07:52.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:52.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj14 testdata.516403 2026-02-22T21:07:52.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:675: TEST_pg_dump_objects_scrubbed: for i in `seq 1 $objects` 2026-02-22T21:07:52.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:677: TEST_pg_dump_objects_scrubbed: rados -p test put obj15 testdata.516403 2026-02-22T21:07:52.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:679: TEST_pg_dump_objects_scrubbed: rm -f testdata.516403 2026-02-22T21:07:52.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:681: TEST_pg_dump_objects_scrubbed: local pgid=1.0 2026-02-22T21:07:52.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:683: TEST_pg_dump_objects_scrubbed: pg_scrub 1.0 2026-02-22T21:07:52.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-22T21:07:52.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-22T21:07:52.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-22T21:07:52.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T21:07:52.264 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:07:52.265 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:07:52.265 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:07:52.265 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:07:52.265 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:07:52.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T21:07:52.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T21:07:52.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T21:07:52.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T21:07:52.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:07:52.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:07:52.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:07:52.712 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:07:52.712 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:07:52.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:07:52.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:07:52.712 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:07:52.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836497 2026-02-22T21:07:52.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836497 2026-02-22T21:07:52.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497' 2026-02-22T21:07:52.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:07:52.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:07:52.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705677 2026-02-22T21:07:52.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705677 2026-02-22T21:07:52.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705677' 2026-02-22T21:07:52.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:07:52.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:07:52.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-22T21:07:52.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-22T21:07:52.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836497 1-38654705677 2-60129542153' 2026-02-22T21:07:52.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:07:52.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836497 2026-02-22T21:07:52.929 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:07:52.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:07:52.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836497 2026-02-22T21:07:52.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:07:52.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836497 2026-02-22T21:07:52.933 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836497 2026-02-22T21:07:52.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-22T21:07:52.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:07:53.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836495 -lt 21474836497 2026-02-22T21:07:53.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:07:54.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:07:54.088 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:07:54.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836498 -lt 21474836497 2026-02-22T21:07:54.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:07:54.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705677 2026-02-22T21:07:54.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:07:54.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:07:54.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705677 2026-02-22T21:07:54.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:07:54.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705677 2026-02-22T21:07:54.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705677' 2026-02-22T21:07:54.252 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705677 2026-02-22T21:07:54.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:07:54.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705678 -lt 38654705677 2026-02-22T21:07:54.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:07:54.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-22T21:07:54.414 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:07:54.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:07:54.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542153 2026-02-22T21:07:54.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:07:54.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542153 2026-02-22T21:07:54.417 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542153 2026-02-22T21:07:54.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-22T21:07:54.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:07:54.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542154 -lt 60129542153 2026-02-22T21:07:54.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T21:07:54.581 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 1.0 loop 0 2026-02-22T21:07:54.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-22T21:07:54.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T21:07:54.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-22T21:07:54.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T21:07:54.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-22T21:07:54.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T21:07:54.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T21:07:54.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T21:07:54.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T21:07:54.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T21:07:54.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-22T21:07:54.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:07:54.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:07:54.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:07:54.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:07:54.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-22T21:07:46.859000+0000 2026-02-22T21:07:54.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-22T21:07:54.950 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0 on osd.1 to scrub 2026-02-22T21:07:54.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-22T21:07:46.859000+0000 2026-02-22T21:07:54.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T21:07:54.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T21:07:46.859000+0000 2026-02-22T21:07:54.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T21:07:54.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T21:07:54.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:07:54.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:07:54.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:07:54.962 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:07:54.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:07:54.963 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:07:55.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T21:07:46.859000+0000 '>' 2026-02-22T21:07:46.859000+0000 2026-02-22T21:07:55.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:07:56.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:07:56.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:07:56.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:07:56.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:07:56.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:07:56.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:07:56.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:07:56.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T21:07:55.260772+0000 '>' 2026-02-22T21:07:46.859000+0000 2026-02-22T21:07:56.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T21:07:56.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:684: TEST_pg_dump_objects_scrubbed: ceph pg 1.0 query 2026-02-22T21:07:56.274 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:684: TEST_pg_dump_objects_scrubbed: jq .info.stats.objects_scrubbed 2026-02-22T21:07:56.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:684: TEST_pg_dump_objects_scrubbed: test 15 = 15 2026-02-22T21:07:56.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:685: TEST_pg_dump_objects_scrubbed: perf_counters td/osd-scrub-test 3 2026-02-22T21:07:56.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-22T21:07:56.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-22T21:07:56.348 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-22T21:07:56.350 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-22T21:07:56.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:07:56.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-22T21:07:56.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:07:56.430 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:07:56.431 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:07:56.431 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:56.431 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:56.431 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:07:56.431 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:07:56.431 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:56.431 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:56.431 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:56.431 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:56.431 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:56.431 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:56.432 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:56.432 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.432 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.432 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:56.432 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:07:56.432 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:56.432 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:56.432 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:07:56.432 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:07:56.432 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:56.432 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:56.432 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:07:56.433 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:56.434 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:56.434 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:56.434 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:56.434 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:56.434 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:56.434 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:56.434 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.434 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.434 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:56.434 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:07:56.434 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:56.434 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:56.435 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:07:56.435 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:07:56.435 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:56.435 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:56.435 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:56.435 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:56.435 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:56.435 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:56.435 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:56.435 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.436 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.436 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:07:56.436 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:56.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:07:56.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-22T21:07:56.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:07:56.502 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:07:56.503 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:07:56.503 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:56.503 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:56.503 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:07:56.503 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:07:56.503 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:56.504 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:56.504 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:56.504 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:56.504 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:56.504 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:56.504 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:56.504 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.504 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.505 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:56.505 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:07:56.505 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:56.505 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:56.505 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:07:56.505 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:07:56.505 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:56.505 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:56.506 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:56.506 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:56.506 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:56.506 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:56.506 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:56.506 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.506 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.506 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:56.506 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:07:56.506 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.507 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:56.508 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:07:56.508 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:56.508 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:56.508 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:07:56.508 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:07:56.508 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:56.508 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:56.508 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:56.508 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 1, 2026-02-22T21:07:56.508 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:56.508 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:56.508 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:56.509 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.509 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.509 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:07:56.509 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:56.509 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:07:56.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-22T21:07:56.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:07:56.576 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:07:56.576 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:07:56.577 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:56.577 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:56.577 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:07:56.577 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:07:56.577 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:56.577 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:56.577 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:56.577 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:56.577 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:56.577 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:56.577 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:56.578 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.578 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.578 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:56.578 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:07:56.578 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:56.578 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:56.578 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:07:56.578 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:07:56.578 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:56.578 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:56.578 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:07:56.579 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:07:56.580 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:07:56.581 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:07:56.581 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:07:56.581 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:07:56.581 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:07:56.581 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:07:56.581 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:07:56.581 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:07:56.581 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:07:56.581 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.581 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:07:56.582 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:07:56.582 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:07:56.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:687: TEST_pg_dump_objects_scrubbed: teardown td/osd-scrub-test 2026-02-22T21:07:56.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:07:56.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:07:56.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:07:56.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:07:56.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:07:56.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:07:56.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:07:56.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:07:56.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:07:56.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:07:56.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:07:56.689 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:07:56.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:07:56.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:07:56.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:07:56.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:07:56.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:07:56.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:07:56.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:07:56.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:07:56.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:07:56.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:07:56.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:07:56.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:56.706 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:56.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:07:56.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:07:56.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:07:56.708 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Teardown Test TEST_pg_dump_objects_scrubbed ------------------ 2026-02-22T21:07:56.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_pg_dump_objects_scrubbed ------------------' 2026-02-22T21:07:56.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-22T21:07:56.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:07:56.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:07:56.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:07:56.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:07:56.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:07:56.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:07:56.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:07:56.710 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:07:56.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:07:56.712 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:07:56.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:07:56.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:07:56.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:07:56.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:07:56.714 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:07:56.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:07:56.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:07:56.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:07:56.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:07:56.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:07:56.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:07:56.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:07:56.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:07:56.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:56.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:56.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:07:56.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:07:56.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:07:56.721 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Complete Test TEST_pg_dump_objects_scrubbed ------------------ 2026-02-22T21:07:56.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_pg_dump_objects_scrubbed ------------------' 2026-02-22T21:07:56.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-22T21:07:56.721 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Prepare Test TEST_scrub_abort ------------------- 2026-02-22T21:07:56.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_scrub_abort -------------------' 2026-02-22T21:07:56.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-22T21:07:56.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-22T21:07:56.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-22T21:07:56.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:07:56.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:07:56.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:07:56.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:07:56.722 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:07:56.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:07:56.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:07:56.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:07:56.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:07:56.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:07:56.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:07:56.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:07:56.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:07:56.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:07:56.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:07:56.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:07:56.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:07:56.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:07:56.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:07:56.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:07:56.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:07:56.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:07:56.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:07:56.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:56.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:56.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:07:56.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:07:56.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:07:56.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-22T21:07:56.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T21:07:56.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:56.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:56.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.516403 2026-02-22T21:07:56.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T21:07:56.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:07:56.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T21:07:56.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-22T21:07:56.739 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Run Test TEST_scrub_abort ----------------------- 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_scrub_abort -----------------------' 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_scrub_abort td/osd-scrub-test 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:404: TEST_scrub_abort: local dir=td/osd-scrub-test 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:405: TEST_scrub_abort: _scrub_abort td/osd-scrub-test scrub 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:293: _scrub_abort: local dir=td/osd-scrub-test 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:294: _scrub_abort: local poolname=test 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:295: _scrub_abort: local OSDS=3 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:296: _scrub_abort: local objects=1000 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:297: _scrub_abort: local type=scrub 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:299: _scrub_abort: TESTDATA=testdata.516403 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:300: _scrub_abort: test scrub = scrub 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:302: _scrub_abort: stopscrub=noscrub 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:303: _scrub_abort: check=noscrub 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:309: _scrub_abort: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-22T21:07:56.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T21:07:56.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T21:07:56.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T21:07:56.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-22T21:07:56.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-22T21:07:56.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T21:07:56.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:07:56.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:07:56.761 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:07:56.761 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:56.761 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:56.761 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:56.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-22T21:07:56.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T21:07:56.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T21:07:56.790 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:07:56.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:07:56.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T21:07:56.791 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T21:07:56.792 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:07:56.792 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:07:56.792 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:07:56.793 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:07:56.793 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:56.793 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:56.793 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T21:07:56.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:07:56.793 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get fsid 2026-02-22T21:07:56.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T21:07:56.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:07:56.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:07:56.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T21:07:56.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T21:07:56.850 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:07:56.850 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:07:56.850 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:07:56.850 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:07:56.850 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:56.850 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:56.850 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T21:07:56.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:07:56.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get mon_host 2026-02-22T21:07:56.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:310: _scrub_abort: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-22T21:07:56.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-22T21:07:56.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T21:07:56.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T21:07:56.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T21:07:56.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-22T21:07:56.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:07:57.000 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T21:07:57.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:07:57.001 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:07:57.001 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:07:57.001 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:57.001 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:57.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:57.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:07:57.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-22T21:07:57.020 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: expr 3 - 1 2026-02-22T21:07:57.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: seq 0 2 2026-02-22T21:07:57.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:07:57.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 0 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-22T21:07:57.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:07:57.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:07:57.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:07:57.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:07:57.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:07:57.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:07:57.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:07:57.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-22T21:07:57.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-22T21:07:57.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:07:57.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=17e3ebcb-0a47-41f5-acd4-1b4289a5d443 2026-02-22T21:07:57.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 17e3ebcb-0a47-41f5-acd4-1b4289a5d443' 2026-02-22T21:07:57.029 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 17e3ebcb-0a47-41f5-acd4-1b4289a5d443 2026-02-22T21:07:57.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:07:57.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAtcJtpl7vZARAArF+HWOLqvRlies90aVwSWw== 2026-02-22T21:07:57.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAtcJtpl7vZARAArF+HWOLqvRlies90aVwSWw=="}' 2026-02-22T21:07:57.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 17e3ebcb-0a47-41f5-acd4-1b4289a5d443 -i td/osd-scrub-test/0/new.json 2026-02-22T21:07:57.125 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:07:57.133 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-22T21:07:57.134 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQAtcJtpl7vZARAArF+HWOLqvRlies90aVwSWw== --osd-uuid 17e3ebcb-0a47-41f5-acd4-1b4289a5d443 2026-02-22T21:07:57.151 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:57.143+0000 7f60c7e6aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:57.152 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:57.143+0000 7f60c7e6aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:57.153 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:57.147+0000 7f60c7e6aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:57.154 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:57.147+0000 7f60c7e6aa40 -1 bdev(0x55ce0e990c00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:07:57.154 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:57.147+0000 7f60c7e6aa40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-22T21:07:57.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-22T21:07:57.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:07:57.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:07:57.949 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T21:07:57.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:07:58.157 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T21:07:58.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T21:07:58.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-22T21:07:58.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:07:58.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:07:58.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:07:58.182 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:58.175+0000 7f1863525a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:58.184 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:58.175+0000 7f1863525a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:58.185 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:58.179+0000 7f1863525a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:58.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T21:07:58.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:07:58.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T21:07:58.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:07:58.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:07:58.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:07:58.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:07:58.317 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:07:58.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:07:58.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:07:58.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:07:58.586 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:58.579+0000 7f1863525a40 -1 Falling back to public interface 2026-02-22T21:07:59.171 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:59.163+0000 7f1863525a40 -1 osd.0 0 log_to_monitors true 2026-02-22T21:07:59.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:07:59.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:07:59.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:07:59.482 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:07:59.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:07:59.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:07:59.641 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2054185984,v1:127.0.0.1:6803/2054185984] [v2:127.0.0.1:6804/2054185984,v1:127.0.0.1:6805/2054185984] exists,up 17e3ebcb-0a47-41f5-acd4-1b4289a5d443 2026-02-22T21:07:59.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:07:59.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:07:59.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:07:59.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:07:59.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 1 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-22T21:07:59.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:07:59.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:07:59.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:07:59.642 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:07:59.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:59.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:07:59.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:07:59.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:07:59.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:07:59.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:07:59.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:07:59.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:07:59.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:07:59.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:07:59.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:07:59.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:07:59.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-22T21:07:59.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-22T21:07:59.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:07:59.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ed34eace-61f3-4a27-bd32-474d45602b81 2026-02-22T21:07:59.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ed34eace-61f3-4a27-bd32-474d45602b81' 2026-02-22T21:07:59.647 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 ed34eace-61f3-4a27-bd32-474d45602b81 2026-02-22T21:07:59.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:07:59.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAvcJtp54PfJhAATsjZLvMBnFaBCRIrdkJHaQ== 2026-02-22T21:07:59.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAvcJtp54PfJhAATsjZLvMBnFaBCRIrdkJHaQ=="}' 2026-02-22T21:07:59.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ed34eace-61f3-4a27-bd32-474d45602b81 -i td/osd-scrub-test/1/new.json 2026-02-22T21:07:59.818 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:07:59.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-22T21:07:59.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQAvcJtp54PfJhAATsjZLvMBnFaBCRIrdkJHaQ== --osd-uuid ed34eace-61f3-4a27-bd32-474d45602b81 2026-02-22T21:07:59.846 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:59.839+0000 7fd42e6cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:59.848 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:59.839+0000 7fd42e6cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:59.849 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:59.843+0000 7fd42e6cba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:07:59.849 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:59.843+0000 7fd42e6cba40 -1 bdev(0x55fa92aeec00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:07:59.849 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:07:59.843+0000 7fd42e6cba40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-22T21:08:00.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-22T21:08:00.500 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:08:00.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:08:00.502 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T21:08:00.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:08:00.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T21:08:00.707 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T21:08:00.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-22T21:08:00.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:08:00.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:08:00.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:08:00.732 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:00.723+0000 7fbff9205a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:08:00.734 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:00.727+0000 7fbff9205a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:08:00.735 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:00.727+0000 7fbff9205a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:08:00.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T21:08:00.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:08:00.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T21:08:00.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:08:00.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:08:00.864 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:08:00.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:08:00.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:08:00.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:08:00.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:08:01.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:08:01.086 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:01.079+0000 7fbff9205a40 -1 Falling back to public interface 2026-02-22T21:08:01.735 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:01.727+0000 7fbff9205a40 -1 osd.1 0 log_to_monitors true 2026-02-22T21:08:02.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:08:02.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:08:02.024 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:08:02.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:08:02.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:08:02.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:08:02.185 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1161564184,v1:127.0.0.1:6811/1161564184] [v2:127.0.0.1:6812/1161564184,v1:127.0.0.1:6813/1161564184] exists,up ed34eace-61f3-4a27-bd32-474d45602b81 2026-02-22T21:08:02.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:08:02.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:08:02.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:08:02.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:311: _scrub_abort: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:08:02.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:317: _scrub_abort: run_osd td/osd-scrub-test 2 --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-22T21:08:02.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:08:02.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:08:02.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T21:08:02.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:08:02.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-22T21:08:02.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:08:02.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:08:02.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:08:02.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:08:02.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-22T21:08:02.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-22T21:08:02.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:08:02.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:08:02.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:08:02.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:08:02.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:08:02.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:08:02.190 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:08:02.190 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:08:02.190 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:08:02.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:08:02.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:08:02.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:08:02.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:08:02.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:08:02.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:08:02.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:08:02.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:08:02.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:08:02.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:08:02.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:08:02.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:08:02.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq' 2026-02-22T21:08:02.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-22T21:08:02.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:08:02.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b8cd0d1f-ab01-4b32-b064-805842603f91 2026-02-22T21:08:02.192 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 b8cd0d1f-ab01-4b32-b064-805842603f91 2026-02-22T21:08:02.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 b8cd0d1f-ab01-4b32-b064-805842603f91' 2026-02-22T21:08:02.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:08:02.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAycJtpWhmoCxAAuneQxtQDdl5BMIY5D7An4A== 2026-02-22T21:08:02.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAycJtpWhmoCxAAuneQxtQDdl5BMIY5D7An4A=="}' 2026-02-22T21:08:02.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b8cd0d1f-ab01-4b32-b064-805842603f91 -i td/osd-scrub-test/2/new.json 2026-02-22T21:08:02.355 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:08:02.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-22T21:08:02.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq --mkfs --key AQAycJtpWhmoCxAAuneQxtQDdl5BMIY5D7An4A== --osd-uuid b8cd0d1f-ab01-4b32-b064-805842603f91 2026-02-22T21:08:02.381 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:02.375+0000 7fddb5e8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:08:02.383 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:02.375+0000 7fddb5e8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:08:02.384 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:02.375+0000 7fddb5e8fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:08:02.384 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:02.379+0000 7fddb5e8fa40 -1 bdev(0x560b9d960c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:08:02.385 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:02.379+0000 7fddb5e8fa40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-22T21:08:03.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-22T21:08:03.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:08:03.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:08:03.032 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T21:08:03.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:08:03.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T21:08:03.225 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T21:08:03.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_sleep=5.0 --osd_scrub_interval_randomize_ratio=0 --osd_op_queue=wpq 2026-02-22T21:08:03.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:08:03.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:08:03.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:08:03.242 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:03.235+0000 7fd3a6b1ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:08:03.244 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:03.235+0000 7fd3a6b1ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:08:03.245 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:03.239+0000 7fd3a6b1ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:08:03.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T21:08:03.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:08:03.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T21:08:03.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:08:03.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:08:03.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:08:03.382 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:08:03.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:08:03.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:08:03.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:08:03.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:08:03.670 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:03.663+0000 7fd3a6b1ba40 -1 Falling back to public interface 2026-02-22T21:08:04.262 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:08:04.255+0000 7fd3a6b1ba40 -1 osd.2 0 log_to_monitors true 2026-02-22T21:08:04.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:08:04.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:08:04.546 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:08:04.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:08:04.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:08:04.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:08:04.705 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3374083623,v1:127.0.0.1:6819/3374083623] [v2:127.0.0.1:6820/3374083623,v1:127.0.0.1:6821/3374083623] exists,up b8cd0d1f-ab01-4b32-b064-805842603f91 2026-02-22T21:08:04.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:08:04.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:08:04.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:08:04.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:325: _scrub_abort: create_pool test 1 1 2026-02-22T21:08:04.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-22T21:08:04.924 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T21:08:04.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T21:08:05.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:326: _scrub_abort: wait_for_clean 2026-02-22T21:08:05.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:08:05.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:08:05.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:08:05.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:08:05.940 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:08:05.940 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:08:05.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:08:05.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:08:05.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:08:06.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:08:06.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:08:06.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:08:06.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:08:06.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:06.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:06.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:06.213 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:06.213 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:06.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:06.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:06.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:06.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836487 2026-02-22T21:08:06.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836487 2026-02-22T21:08:06.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487' 2026-02-22T21:08:06.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:06.287 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:06.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705669 2026-02-22T21:08:06.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705669 2026-02-22T21:08:06.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-38654705669' 2026-02-22T21:08:06.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:06.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:06.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-22T21:08:06.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-22T21:08:06.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836487 1-38654705669 2-60129542147' 2026-02-22T21:08:06.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:06.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836487 2026-02-22T21:08:06.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:06.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:06.434 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836487 2026-02-22T21:08:06.434 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:06.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836487 2026-02-22T21:08:06.436 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836487 2026-02-22T21:08:06.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-22T21:08:06.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:06.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836487 2026-02-22T21:08:06.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:07.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:08:07.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:07.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836487 2026-02-22T21:08:07.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:07.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705669 2026-02-22T21:08:07.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:07.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:07.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705669 2026-02-22T21:08:07.766 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:07.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705669 2026-02-22T21:08:07.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2026-02-22T21:08:07.767 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705669 2026-02-22T21:08:07.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:07.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705670 -lt 38654705669 2026-02-22T21:08:07.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:07.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-22T21:08:07.936 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:07.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:07.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542147 2026-02-22T21:08:07.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:07.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542147 2026-02-22T21:08:07.940 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542147 2026-02-22T21:08:07.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-22T21:08:07.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:08.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542148 -lt 60129542147 2026-02-22T21:08:08.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:08:08.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:08:08.110 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:08:08.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:08:08.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:08:08.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:08:08.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:08:08.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:08:08.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:08:08.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:08:08.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:08:08.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:08:08.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:08:08.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:08:08.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:08:08.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:08:08.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:08:08.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:08:08.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: ceph osd dump 2026-02-22T21:08:08.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: grep '^pool.*['\'']test['\'']' 2026-02-22T21:08:08.650 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: awk '{ print $2 }' 2026-02-22T21:08:08.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:327: _scrub_abort: poolid=1 2026-02-22T21:08:08.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:329: _scrub_abort: dd if=/dev/urandom of=testdata.516403 bs=1032 count=1 2026-02-22T21:08:08.807 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T21:08:08.807 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T21:08:08.807 INFO:tasks.workunit.client.0.trial150.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 3.7109e-05 s, 27.8 MB/s 2026-02-22T21:08:08.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: seq 1 1000 2026-02-22T21:08:08.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:08.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj1 testdata.516403 2026-02-22T21:08:08.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:08.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj2 testdata.516403 2026-02-22T21:08:08.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:08.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj3 testdata.516403 2026-02-22T21:08:08.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:08.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj4 testdata.516403 2026-02-22T21:08:08.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:08.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj5 testdata.516403 2026-02-22T21:08:08.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:08.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj6 testdata.516403 2026-02-22T21:08:08.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:08.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj7 testdata.516403 2026-02-22T21:08:08.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:08.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj8 testdata.516403 2026-02-22T21:08:08.973 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:08.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj9 testdata.516403 2026-02-22T21:08:08.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:08.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj10 testdata.516403 2026-02-22T21:08:09.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj11 testdata.516403 2026-02-22T21:08:09.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj12 testdata.516403 2026-02-22T21:08:09.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj13 testdata.516403 2026-02-22T21:08:09.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj14 testdata.516403 2026-02-22T21:08:09.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj15 testdata.516403 2026-02-22T21:08:09.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj16 testdata.516403 2026-02-22T21:08:09.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj17 testdata.516403 2026-02-22T21:08:09.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj18 testdata.516403 2026-02-22T21:08:09.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj19 testdata.516403 2026-02-22T21:08:09.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj20 testdata.516403 2026-02-22T21:08:09.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj21 testdata.516403 2026-02-22T21:08:09.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj22 testdata.516403 2026-02-22T21:08:09.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj23 testdata.516403 2026-02-22T21:08:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj24 testdata.516403 2026-02-22T21:08:09.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj25 testdata.516403 2026-02-22T21:08:09.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj26 testdata.516403 2026-02-22T21:08:09.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj27 testdata.516403 2026-02-22T21:08:09.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj28 testdata.516403 2026-02-22T21:08:09.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj29 testdata.516403 2026-02-22T21:08:09.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj30 testdata.516403 2026-02-22T21:08:09.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj31 testdata.516403 2026-02-22T21:08:09.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj32 testdata.516403 2026-02-22T21:08:09.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj33 testdata.516403 2026-02-22T21:08:09.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj34 testdata.516403 2026-02-22T21:08:09.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj35 testdata.516403 2026-02-22T21:08:09.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj36 testdata.516403 2026-02-22T21:08:09.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj37 testdata.516403 2026-02-22T21:08:09.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.628 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj38 testdata.516403 2026-02-22T21:08:09.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj39 testdata.516403 2026-02-22T21:08:09.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj40 testdata.516403 2026-02-22T21:08:09.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj41 testdata.516403 2026-02-22T21:08:09.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj42 testdata.516403 2026-02-22T21:08:09.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj43 testdata.516403 2026-02-22T21:08:09.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj44 testdata.516403 2026-02-22T21:08:09.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj45 testdata.516403 2026-02-22T21:08:09.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj46 testdata.516403 2026-02-22T21:08:09.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj47 testdata.516403 2026-02-22T21:08:09.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj48 testdata.516403 2026-02-22T21:08:09.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj49 testdata.516403 2026-02-22T21:08:09.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj50 testdata.516403 2026-02-22T21:08:09.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj51 testdata.516403 2026-02-22T21:08:09.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj52 testdata.516403 2026-02-22T21:08:09.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:09.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj53 testdata.516403 2026-02-22T21:08:10.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj54 testdata.516403 2026-02-22T21:08:10.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj55 testdata.516403 2026-02-22T21:08:10.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj56 testdata.516403 2026-02-22T21:08:10.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj57 testdata.516403 2026-02-22T21:08:10.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj58 testdata.516403 2026-02-22T21:08:10.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj59 testdata.516403 2026-02-22T21:08:10.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj60 testdata.516403 2026-02-22T21:08:10.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj61 testdata.516403 2026-02-22T21:08:10.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj62 testdata.516403 2026-02-22T21:08:10.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj63 testdata.516403 2026-02-22T21:08:10.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj64 testdata.516403 2026-02-22T21:08:10.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj65 testdata.516403 2026-02-22T21:08:10.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj66 testdata.516403 2026-02-22T21:08:10.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj67 testdata.516403 2026-02-22T21:08:10.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj68 testdata.516403 2026-02-22T21:08:10.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj69 testdata.516403 2026-02-22T21:08:10.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj70 testdata.516403 2026-02-22T21:08:10.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj71 testdata.516403 2026-02-22T21:08:10.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj72 testdata.516403 2026-02-22T21:08:10.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj73 testdata.516403 2026-02-22T21:08:10.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.460 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj74 testdata.516403 2026-02-22T21:08:10.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj75 testdata.516403 2026-02-22T21:08:10.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj76 testdata.516403 2026-02-22T21:08:10.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj77 testdata.516403 2026-02-22T21:08:10.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj78 testdata.516403 2026-02-22T21:08:10.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj79 testdata.516403 2026-02-22T21:08:10.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj80 testdata.516403 2026-02-22T21:08:10.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj81 testdata.516403 2026-02-22T21:08:10.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj82 testdata.516403 2026-02-22T21:08:10.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj83 testdata.516403 2026-02-22T21:08:10.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.706 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj84 testdata.516403 2026-02-22T21:08:10.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj85 testdata.516403 2026-02-22T21:08:10.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj86 testdata.516403 2026-02-22T21:08:10.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj87 testdata.516403 2026-02-22T21:08:10.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj88 testdata.516403 2026-02-22T21:08:10.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj89 testdata.516403 2026-02-22T21:08:10.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj90 testdata.516403 2026-02-22T21:08:10.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj91 testdata.516403 2026-02-22T21:08:10.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj92 testdata.516403 2026-02-22T21:08:10.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj93 testdata.516403 2026-02-22T21:08:10.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj94 testdata.516403 2026-02-22T21:08:10.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj95 testdata.516403 2026-02-22T21:08:10.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:10.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj96 testdata.516403 2026-02-22T21:08:11.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj97 testdata.516403 2026-02-22T21:08:11.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj98 testdata.516403 2026-02-22T21:08:11.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.064 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj99 testdata.516403 2026-02-22T21:08:11.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj100 testdata.516403 2026-02-22T21:08:11.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj101 testdata.516403 2026-02-22T21:08:11.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj102 testdata.516403 2026-02-22T21:08:11.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj103 testdata.516403 2026-02-22T21:08:11.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj104 testdata.516403 2026-02-22T21:08:11.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj105 testdata.516403 2026-02-22T21:08:11.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj106 testdata.516403 2026-02-22T21:08:11.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj107 testdata.516403 2026-02-22T21:08:11.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj108 testdata.516403 2026-02-22T21:08:11.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj109 testdata.516403 2026-02-22T21:08:11.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj110 testdata.516403 2026-02-22T21:08:11.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj111 testdata.516403 2026-02-22T21:08:11.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj112 testdata.516403 2026-02-22T21:08:11.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj113 testdata.516403 2026-02-22T21:08:11.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj114 testdata.516403 2026-02-22T21:08:11.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj115 testdata.516403 2026-02-22T21:08:11.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj116 testdata.516403 2026-02-22T21:08:11.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj117 testdata.516403 2026-02-22T21:08:11.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj118 testdata.516403 2026-02-22T21:08:11.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj119 testdata.516403 2026-02-22T21:08:11.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj120 testdata.516403 2026-02-22T21:08:11.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj121 testdata.516403 2026-02-22T21:08:11.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj122 testdata.516403 2026-02-22T21:08:11.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj123 testdata.516403 2026-02-22T21:08:11.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj124 testdata.516403 2026-02-22T21:08:11.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj125 testdata.516403 2026-02-22T21:08:11.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj126 testdata.516403 2026-02-22T21:08:11.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj127 testdata.516403 2026-02-22T21:08:11.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj128 testdata.516403 2026-02-22T21:08:11.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj129 testdata.516403 2026-02-22T21:08:11.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj130 testdata.516403 2026-02-22T21:08:11.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj131 testdata.516403 2026-02-22T21:08:11.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj132 testdata.516403 2026-02-22T21:08:11.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj133 testdata.516403 2026-02-22T21:08:11.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj134 testdata.516403 2026-02-22T21:08:11.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj135 testdata.516403 2026-02-22T21:08:11.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj136 testdata.516403 2026-02-22T21:08:11.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj137 testdata.516403 2026-02-22T21:08:11.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj138 testdata.516403 2026-02-22T21:08:11.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:11.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj139 testdata.516403 2026-02-22T21:08:12.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj140 testdata.516403 2026-02-22T21:08:12.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj141 testdata.516403 2026-02-22T21:08:12.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.065 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj142 testdata.516403 2026-02-22T21:08:12.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj143 testdata.516403 2026-02-22T21:08:12.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj144 testdata.516403 2026-02-22T21:08:12.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj145 testdata.516403 2026-02-22T21:08:12.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj146 testdata.516403 2026-02-22T21:08:12.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj147 testdata.516403 2026-02-22T21:08:12.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj148 testdata.516403 2026-02-22T21:08:12.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj149 testdata.516403 2026-02-22T21:08:12.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj150 testdata.516403 2026-02-22T21:08:12.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj151 testdata.516403 2026-02-22T21:08:12.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj152 testdata.516403 2026-02-22T21:08:12.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj153 testdata.516403 2026-02-22T21:08:12.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj154 testdata.516403 2026-02-22T21:08:12.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj155 testdata.516403 2026-02-22T21:08:12.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj156 testdata.516403 2026-02-22T21:08:12.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj157 testdata.516403 2026-02-22T21:08:12.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj158 testdata.516403 2026-02-22T21:08:12.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.467 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj159 testdata.516403 2026-02-22T21:08:12.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj160 testdata.516403 2026-02-22T21:08:12.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj161 testdata.516403 2026-02-22T21:08:12.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj162 testdata.516403 2026-02-22T21:08:12.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj163 testdata.516403 2026-02-22T21:08:12.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj164 testdata.516403 2026-02-22T21:08:12.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj165 testdata.516403 2026-02-22T21:08:12.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj166 testdata.516403 2026-02-22T21:08:12.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj167 testdata.516403 2026-02-22T21:08:12.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj168 testdata.516403 2026-02-22T21:08:12.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj169 testdata.516403 2026-02-22T21:08:12.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj170 testdata.516403 2026-02-22T21:08:12.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj171 testdata.516403 2026-02-22T21:08:12.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj172 testdata.516403 2026-02-22T21:08:12.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.789 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj173 testdata.516403 2026-02-22T21:08:12.813 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj174 testdata.516403 2026-02-22T21:08:12.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.837 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj175 testdata.516403 2026-02-22T21:08:12.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj176 testdata.516403 2026-02-22T21:08:12.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj177 testdata.516403 2026-02-22T21:08:12.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj178 testdata.516403 2026-02-22T21:08:12.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj179 testdata.516403 2026-02-22T21:08:12.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj180 testdata.516403 2026-02-22T21:08:12.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:12.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj181 testdata.516403 2026-02-22T21:08:13.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj182 testdata.516403 2026-02-22T21:08:13.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj183 testdata.516403 2026-02-22T21:08:13.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj184 testdata.516403 2026-02-22T21:08:13.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj185 testdata.516403 2026-02-22T21:08:13.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj186 testdata.516403 2026-02-22T21:08:13.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj187 testdata.516403 2026-02-22T21:08:13.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj188 testdata.516403 2026-02-22T21:08:13.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj189 testdata.516403 2026-02-22T21:08:13.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj190 testdata.516403 2026-02-22T21:08:13.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj191 testdata.516403 2026-02-22T21:08:13.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj192 testdata.516403 2026-02-22T21:08:13.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj193 testdata.516403 2026-02-22T21:08:13.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj194 testdata.516403 2026-02-22T21:08:13.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj195 testdata.516403 2026-02-22T21:08:13.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj196 testdata.516403 2026-02-22T21:08:13.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.359 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj197 testdata.516403 2026-02-22T21:08:13.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj198 testdata.516403 2026-02-22T21:08:13.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj199 testdata.516403 2026-02-22T21:08:13.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj200 testdata.516403 2026-02-22T21:08:13.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj201 testdata.516403 2026-02-22T21:08:13.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj202 testdata.516403 2026-02-22T21:08:13.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj203 testdata.516403 2026-02-22T21:08:13.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj204 testdata.516403 2026-02-22T21:08:13.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj205 testdata.516403 2026-02-22T21:08:13.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj206 testdata.516403 2026-02-22T21:08:13.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj207 testdata.516403 2026-02-22T21:08:13.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj208 testdata.516403 2026-02-22T21:08:13.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj209 testdata.516403 2026-02-22T21:08:13.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj210 testdata.516403 2026-02-22T21:08:13.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj211 testdata.516403 2026-02-22T21:08:13.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj212 testdata.516403 2026-02-22T21:08:13.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj213 testdata.516403 2026-02-22T21:08:13.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj214 testdata.516403 2026-02-22T21:08:13.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj215 testdata.516403 2026-02-22T21:08:13.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj216 testdata.516403 2026-02-22T21:08:13.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj217 testdata.516403 2026-02-22T21:08:13.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj218 testdata.516403 2026-02-22T21:08:13.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj219 testdata.516403 2026-02-22T21:08:13.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj220 testdata.516403 2026-02-22T21:08:13.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj221 testdata.516403 2026-02-22T21:08:13.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj222 testdata.516403 2026-02-22T21:08:13.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj223 testdata.516403 2026-02-22T21:08:13.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:13.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj224 testdata.516403 2026-02-22T21:08:14.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj225 testdata.516403 2026-02-22T21:08:14.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj226 testdata.516403 2026-02-22T21:08:14.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj227 testdata.516403 2026-02-22T21:08:14.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj228 testdata.516403 2026-02-22T21:08:14.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj229 testdata.516403 2026-02-22T21:08:14.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj230 testdata.516403 2026-02-22T21:08:14.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj231 testdata.516403 2026-02-22T21:08:14.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj232 testdata.516403 2026-02-22T21:08:14.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj233 testdata.516403 2026-02-22T21:08:14.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj234 testdata.516403 2026-02-22T21:08:14.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj235 testdata.516403 2026-02-22T21:08:14.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj236 testdata.516403 2026-02-22T21:08:14.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.300 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj237 testdata.516403 2026-02-22T21:08:14.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj238 testdata.516403 2026-02-22T21:08:14.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj239 testdata.516403 2026-02-22T21:08:14.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj240 testdata.516403 2026-02-22T21:08:14.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj241 testdata.516403 2026-02-22T21:08:14.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj242 testdata.516403 2026-02-22T21:08:14.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj243 testdata.516403 2026-02-22T21:08:14.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj244 testdata.516403 2026-02-22T21:08:14.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.488 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj245 testdata.516403 2026-02-22T21:08:14.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj246 testdata.516403 2026-02-22T21:08:14.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj247 testdata.516403 2026-02-22T21:08:14.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj248 testdata.516403 2026-02-22T21:08:14.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.585 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj249 testdata.516403 2026-02-22T21:08:14.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj250 testdata.516403 2026-02-22T21:08:14.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj251 testdata.516403 2026-02-22T21:08:14.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj252 testdata.516403 2026-02-22T21:08:14.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj253 testdata.516403 2026-02-22T21:08:14.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj254 testdata.516403 2026-02-22T21:08:14.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj255 testdata.516403 2026-02-22T21:08:14.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj256 testdata.516403 2026-02-22T21:08:14.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj257 testdata.516403 2026-02-22T21:08:14.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj258 testdata.516403 2026-02-22T21:08:14.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj259 testdata.516403 2026-02-22T21:08:14.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj260 testdata.516403 2026-02-22T21:08:14.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj261 testdata.516403 2026-02-22T21:08:14.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj262 testdata.516403 2026-02-22T21:08:14.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj263 testdata.516403 2026-02-22T21:08:14.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj264 testdata.516403 2026-02-22T21:08:14.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj265 testdata.516403 2026-02-22T21:08:14.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:14.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj266 testdata.516403 2026-02-22T21:08:15.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj267 testdata.516403 2026-02-22T21:08:15.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj268 testdata.516403 2026-02-22T21:08:15.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj269 testdata.516403 2026-02-22T21:08:15.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj270 testdata.516403 2026-02-22T21:08:15.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj271 testdata.516403 2026-02-22T21:08:15.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj272 testdata.516403 2026-02-22T21:08:15.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj273 testdata.516403 2026-02-22T21:08:15.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj274 testdata.516403 2026-02-22T21:08:15.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj275 testdata.516403 2026-02-22T21:08:15.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj276 testdata.516403 2026-02-22T21:08:15.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj277 testdata.516403 2026-02-22T21:08:15.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj278 testdata.516403 2026-02-22T21:08:15.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj279 testdata.516403 2026-02-22T21:08:15.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj280 testdata.516403 2026-02-22T21:08:15.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj281 testdata.516403 2026-02-22T21:08:15.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj282 testdata.516403 2026-02-22T21:08:15.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj283 testdata.516403 2026-02-22T21:08:15.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj284 testdata.516403 2026-02-22T21:08:15.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj285 testdata.516403 2026-02-22T21:08:15.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj286 testdata.516403 2026-02-22T21:08:15.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj287 testdata.516403 2026-02-22T21:08:15.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj288 testdata.516403 2026-02-22T21:08:15.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj289 testdata.516403 2026-02-22T21:08:15.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj290 testdata.516403 2026-02-22T21:08:15.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.605 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj291 testdata.516403 2026-02-22T21:08:15.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj292 testdata.516403 2026-02-22T21:08:15.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj293 testdata.516403 2026-02-22T21:08:15.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj294 testdata.516403 2026-02-22T21:08:15.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj295 testdata.516403 2026-02-22T21:08:15.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj296 testdata.516403 2026-02-22T21:08:15.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj297 testdata.516403 2026-02-22T21:08:15.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj298 testdata.516403 2026-02-22T21:08:15.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj299 testdata.516403 2026-02-22T21:08:15.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj300 testdata.516403 2026-02-22T21:08:15.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj301 testdata.516403 2026-02-22T21:08:15.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj302 testdata.516403 2026-02-22T21:08:15.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj303 testdata.516403 2026-02-22T21:08:15.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj304 testdata.516403 2026-02-22T21:08:15.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj305 testdata.516403 2026-02-22T21:08:15.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj306 testdata.516403 2026-02-22T21:08:15.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj307 testdata.516403 2026-02-22T21:08:15.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:15.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj308 testdata.516403 2026-02-22T21:08:16.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj309 testdata.516403 2026-02-22T21:08:16.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj310 testdata.516403 2026-02-22T21:08:16.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj311 testdata.516403 2026-02-22T21:08:16.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj312 testdata.516403 2026-02-22T21:08:16.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj313 testdata.516403 2026-02-22T21:08:16.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj314 testdata.516403 2026-02-22T21:08:16.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj315 testdata.516403 2026-02-22T21:08:16.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj316 testdata.516403 2026-02-22T21:08:16.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj317 testdata.516403 2026-02-22T21:08:16.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj318 testdata.516403 2026-02-22T21:08:16.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj319 testdata.516403 2026-02-22T21:08:16.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj320 testdata.516403 2026-02-22T21:08:16.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj321 testdata.516403 2026-02-22T21:08:16.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj322 testdata.516403 2026-02-22T21:08:16.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj323 testdata.516403 2026-02-22T21:08:16.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj324 testdata.516403 2026-02-22T21:08:16.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj325 testdata.516403 2026-02-22T21:08:16.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj326 testdata.516403 2026-02-22T21:08:16.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj327 testdata.516403 2026-02-22T21:08:16.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.482 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj328 testdata.516403 2026-02-22T21:08:16.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj329 testdata.516403 2026-02-22T21:08:16.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj330 testdata.516403 2026-02-22T21:08:16.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj331 testdata.516403 2026-02-22T21:08:16.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj332 testdata.516403 2026-02-22T21:08:16.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj333 testdata.516403 2026-02-22T21:08:16.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj334 testdata.516403 2026-02-22T21:08:16.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj335 testdata.516403 2026-02-22T21:08:16.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj336 testdata.516403 2026-02-22T21:08:16.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj337 testdata.516403 2026-02-22T21:08:16.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj338 testdata.516403 2026-02-22T21:08:16.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj339 testdata.516403 2026-02-22T21:08:16.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj340 testdata.516403 2026-02-22T21:08:16.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.799 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj341 testdata.516403 2026-02-22T21:08:16.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj342 testdata.516403 2026-02-22T21:08:16.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj343 testdata.516403 2026-02-22T21:08:16.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj344 testdata.516403 2026-02-22T21:08:16.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj345 testdata.516403 2026-02-22T21:08:16.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj346 testdata.516403 2026-02-22T21:08:16.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj347 testdata.516403 2026-02-22T21:08:16.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj348 testdata.516403 2026-02-22T21:08:16.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:16.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj349 testdata.516403 2026-02-22T21:08:17.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj350 testdata.516403 2026-02-22T21:08:17.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj351 testdata.516403 2026-02-22T21:08:17.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj352 testdata.516403 2026-02-22T21:08:17.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj353 testdata.516403 2026-02-22T21:08:17.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj354 testdata.516403 2026-02-22T21:08:17.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj355 testdata.516403 2026-02-22T21:08:17.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj356 testdata.516403 2026-02-22T21:08:17.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj357 testdata.516403 2026-02-22T21:08:17.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj358 testdata.516403 2026-02-22T21:08:17.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj359 testdata.516403 2026-02-22T21:08:17.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj360 testdata.516403 2026-02-22T21:08:17.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj361 testdata.516403 2026-02-22T21:08:17.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj362 testdata.516403 2026-02-22T21:08:17.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.331 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj363 testdata.516403 2026-02-22T21:08:17.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj364 testdata.516403 2026-02-22T21:08:17.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj365 testdata.516403 2026-02-22T21:08:17.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj366 testdata.516403 2026-02-22T21:08:17.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj367 testdata.516403 2026-02-22T21:08:17.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj368 testdata.516403 2026-02-22T21:08:17.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.469 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj369 testdata.516403 2026-02-22T21:08:17.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj370 testdata.516403 2026-02-22T21:08:17.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj371 testdata.516403 2026-02-22T21:08:17.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj372 testdata.516403 2026-02-22T21:08:17.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj373 testdata.516403 2026-02-22T21:08:17.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj374 testdata.516403 2026-02-22T21:08:17.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj375 testdata.516403 2026-02-22T21:08:17.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj376 testdata.516403 2026-02-22T21:08:17.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj377 testdata.516403 2026-02-22T21:08:17.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj378 testdata.516403 2026-02-22T21:08:17.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj379 testdata.516403 2026-02-22T21:08:17.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj380 testdata.516403 2026-02-22T21:08:17.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj381 testdata.516403 2026-02-22T21:08:17.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj382 testdata.516403 2026-02-22T21:08:17.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.793 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj383 testdata.516403 2026-02-22T21:08:17.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj384 testdata.516403 2026-02-22T21:08:17.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj385 testdata.516403 2026-02-22T21:08:17.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj386 testdata.516403 2026-02-22T21:08:17.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj387 testdata.516403 2026-02-22T21:08:17.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj388 testdata.516403 2026-02-22T21:08:17.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj389 testdata.516403 2026-02-22T21:08:17.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj390 testdata.516403 2026-02-22T21:08:17.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj391 testdata.516403 2026-02-22T21:08:17.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:17.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj392 testdata.516403 2026-02-22T21:08:18.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj393 testdata.516403 2026-02-22T21:08:18.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj394 testdata.516403 2026-02-22T21:08:18.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj395 testdata.516403 2026-02-22T21:08:18.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj396 testdata.516403 2026-02-22T21:08:18.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj397 testdata.516403 2026-02-22T21:08:18.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj398 testdata.516403 2026-02-22T21:08:18.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj399 testdata.516403 2026-02-22T21:08:18.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj400 testdata.516403 2026-02-22T21:08:18.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj401 testdata.516403 2026-02-22T21:08:18.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj402 testdata.516403 2026-02-22T21:08:18.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj403 testdata.516403 2026-02-22T21:08:18.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj404 testdata.516403 2026-02-22T21:08:18.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj405 testdata.516403 2026-02-22T21:08:18.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj406 testdata.516403 2026-02-22T21:08:18.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj407 testdata.516403 2026-02-22T21:08:18.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj408 testdata.516403 2026-02-22T21:08:18.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj409 testdata.516403 2026-02-22T21:08:18.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj410 testdata.516403 2026-02-22T21:08:18.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj411 testdata.516403 2026-02-22T21:08:18.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj412 testdata.516403 2026-02-22T21:08:18.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj413 testdata.516403 2026-02-22T21:08:18.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj414 testdata.516403 2026-02-22T21:08:18.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj415 testdata.516403 2026-02-22T21:08:18.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj416 testdata.516403 2026-02-22T21:08:18.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj417 testdata.516403 2026-02-22T21:08:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj418 testdata.516403 2026-02-22T21:08:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj419 testdata.516403 2026-02-22T21:08:18.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.652 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj420 testdata.516403 2026-02-22T21:08:18.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj421 testdata.516403 2026-02-22T21:08:18.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj422 testdata.516403 2026-02-22T21:08:18.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj423 testdata.516403 2026-02-22T21:08:18.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj424 testdata.516403 2026-02-22T21:08:18.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj425 testdata.516403 2026-02-22T21:08:18.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.791 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj426 testdata.516403 2026-02-22T21:08:18.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj427 testdata.516403 2026-02-22T21:08:18.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj428 testdata.516403 2026-02-22T21:08:18.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj429 testdata.516403 2026-02-22T21:08:18.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj430 testdata.516403 2026-02-22T21:08:18.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj431 testdata.516403 2026-02-22T21:08:18.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj432 testdata.516403 2026-02-22T21:08:18.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj433 testdata.516403 2026-02-22T21:08:18.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:18.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj434 testdata.516403 2026-02-22T21:08:19.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj435 testdata.516403 2026-02-22T21:08:19.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj436 testdata.516403 2026-02-22T21:08:19.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj437 testdata.516403 2026-02-22T21:08:19.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj438 testdata.516403 2026-02-22T21:08:19.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj439 testdata.516403 2026-02-22T21:08:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj440 testdata.516403 2026-02-22T21:08:19.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj441 testdata.516403 2026-02-22T21:08:19.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.173 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj442 testdata.516403 2026-02-22T21:08:19.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj443 testdata.516403 2026-02-22T21:08:19.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj444 testdata.516403 2026-02-22T21:08:19.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj445 testdata.516403 2026-02-22T21:08:19.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj446 testdata.516403 2026-02-22T21:08:19.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj447 testdata.516403 2026-02-22T21:08:19.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj448 testdata.516403 2026-02-22T21:08:19.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj449 testdata.516403 2026-02-22T21:08:19.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj450 testdata.516403 2026-02-22T21:08:19.388 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj451 testdata.516403 2026-02-22T21:08:19.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj452 testdata.516403 2026-02-22T21:08:19.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj453 testdata.516403 2026-02-22T21:08:19.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj454 testdata.516403 2026-02-22T21:08:19.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj455 testdata.516403 2026-02-22T21:08:19.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj456 testdata.516403 2026-02-22T21:08:19.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj457 testdata.516403 2026-02-22T21:08:19.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.547 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj458 testdata.516403 2026-02-22T21:08:19.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj459 testdata.516403 2026-02-22T21:08:19.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj460 testdata.516403 2026-02-22T21:08:19.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj461 testdata.516403 2026-02-22T21:08:19.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj462 testdata.516403 2026-02-22T21:08:19.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj463 testdata.516403 2026-02-22T21:08:19.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj464 testdata.516403 2026-02-22T21:08:19.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj465 testdata.516403 2026-02-22T21:08:19.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj466 testdata.516403 2026-02-22T21:08:19.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj467 testdata.516403 2026-02-22T21:08:19.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj468 testdata.516403 2026-02-22T21:08:19.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj469 testdata.516403 2026-02-22T21:08:19.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj470 testdata.516403 2026-02-22T21:08:19.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj471 testdata.516403 2026-02-22T21:08:19.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj472 testdata.516403 2026-02-22T21:08:19.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj473 testdata.516403 2026-02-22T21:08:19.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj474 testdata.516403 2026-02-22T21:08:19.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj475 testdata.516403 2026-02-22T21:08:19.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.971 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj476 testdata.516403 2026-02-22T21:08:19.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:19.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj477 testdata.516403 2026-02-22T21:08:20.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj478 testdata.516403 2026-02-22T21:08:20.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj479 testdata.516403 2026-02-22T21:08:20.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj480 testdata.516403 2026-02-22T21:08:20.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj481 testdata.516403 2026-02-22T21:08:20.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj482 testdata.516403 2026-02-22T21:08:20.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.142 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj483 testdata.516403 2026-02-22T21:08:20.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj484 testdata.516403 2026-02-22T21:08:20.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj485 testdata.516403 2026-02-22T21:08:20.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj486 testdata.516403 2026-02-22T21:08:20.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj487 testdata.516403 2026-02-22T21:08:20.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj488 testdata.516403 2026-02-22T21:08:20.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj489 testdata.516403 2026-02-22T21:08:20.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj490 testdata.516403 2026-02-22T21:08:20.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj491 testdata.516403 2026-02-22T21:08:20.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj492 testdata.516403 2026-02-22T21:08:20.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj493 testdata.516403 2026-02-22T21:08:20.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj494 testdata.516403 2026-02-22T21:08:20.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj495 testdata.516403 2026-02-22T21:08:20.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj496 testdata.516403 2026-02-22T21:08:20.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj497 testdata.516403 2026-02-22T21:08:20.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj498 testdata.516403 2026-02-22T21:08:20.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj499 testdata.516403 2026-02-22T21:08:20.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj500 testdata.516403 2026-02-22T21:08:20.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj501 testdata.516403 2026-02-22T21:08:20.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj502 testdata.516403 2026-02-22T21:08:20.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj503 testdata.516403 2026-02-22T21:08:20.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj504 testdata.516403 2026-02-22T21:08:20.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj505 testdata.516403 2026-02-22T21:08:20.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj506 testdata.516403 2026-02-22T21:08:20.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.713 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj507 testdata.516403 2026-02-22T21:08:20.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj508 testdata.516403 2026-02-22T21:08:20.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj509 testdata.516403 2026-02-22T21:08:20.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj510 testdata.516403 2026-02-22T21:08:20.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj511 testdata.516403 2026-02-22T21:08:20.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj512 testdata.516403 2026-02-22T21:08:20.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj513 testdata.516403 2026-02-22T21:08:20.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.885 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj514 testdata.516403 2026-02-22T21:08:20.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj515 testdata.516403 2026-02-22T21:08:20.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj516 testdata.516403 2026-02-22T21:08:20.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj517 testdata.516403 2026-02-22T21:08:20.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:20.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj518 testdata.516403 2026-02-22T21:08:21.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj519 testdata.516403 2026-02-22T21:08:21.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj520 testdata.516403 2026-02-22T21:08:21.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj521 testdata.516403 2026-02-22T21:08:21.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj522 testdata.516403 2026-02-22T21:08:21.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj523 testdata.516403 2026-02-22T21:08:21.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj524 testdata.516403 2026-02-22T21:08:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj525 testdata.516403 2026-02-22T21:08:21.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj526 testdata.516403 2026-02-22T21:08:21.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj527 testdata.516403 2026-02-22T21:08:21.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj528 testdata.516403 2026-02-22T21:08:21.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj529 testdata.516403 2026-02-22T21:08:21.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj530 testdata.516403 2026-02-22T21:08:21.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj531 testdata.516403 2026-02-22T21:08:21.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj532 testdata.516403 2026-02-22T21:08:21.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj533 testdata.516403 2026-02-22T21:08:21.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj534 testdata.516403 2026-02-22T21:08:21.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj535 testdata.516403 2026-02-22T21:08:21.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj536 testdata.516403 2026-02-22T21:08:21.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj537 testdata.516403 2026-02-22T21:08:21.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj538 testdata.516403 2026-02-22T21:08:21.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj539 testdata.516403 2026-02-22T21:08:21.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj540 testdata.516403 2026-02-22T21:08:21.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj541 testdata.516403 2026-02-22T21:08:21.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj542 testdata.516403 2026-02-22T21:08:21.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj543 testdata.516403 2026-02-22T21:08:21.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj544 testdata.516403 2026-02-22T21:08:21.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj545 testdata.516403 2026-02-22T21:08:21.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj546 testdata.516403 2026-02-22T21:08:21.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj547 testdata.516403 2026-02-22T21:08:21.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj548 testdata.516403 2026-02-22T21:08:21.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj549 testdata.516403 2026-02-22T21:08:21.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj550 testdata.516403 2026-02-22T21:08:21.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj551 testdata.516403 2026-02-22T21:08:21.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj552 testdata.516403 2026-02-22T21:08:21.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj553 testdata.516403 2026-02-22T21:08:21.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj554 testdata.516403 2026-02-22T21:08:21.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj555 testdata.516403 2026-02-22T21:08:21.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj556 testdata.516403 2026-02-22T21:08:21.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj557 testdata.516403 2026-02-22T21:08:21.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj558 testdata.516403 2026-02-22T21:08:21.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj559 testdata.516403 2026-02-22T21:08:21.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj560 testdata.516403 2026-02-22T21:08:21.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:21.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj561 testdata.516403 2026-02-22T21:08:22.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj562 testdata.516403 2026-02-22T21:08:22.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj563 testdata.516403 2026-02-22T21:08:22.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj564 testdata.516403 2026-02-22T21:08:22.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj565 testdata.516403 2026-02-22T21:08:22.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj566 testdata.516403 2026-02-22T21:08:22.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj567 testdata.516403 2026-02-22T21:08:22.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj568 testdata.516403 2026-02-22T21:08:22.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj569 testdata.516403 2026-02-22T21:08:22.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.201 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj570 testdata.516403 2026-02-22T21:08:22.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj571 testdata.516403 2026-02-22T21:08:22.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj572 testdata.516403 2026-02-22T21:08:22.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj573 testdata.516403 2026-02-22T21:08:22.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj574 testdata.516403 2026-02-22T21:08:22.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj575 testdata.516403 2026-02-22T21:08:22.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj576 testdata.516403 2026-02-22T21:08:22.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj577 testdata.516403 2026-02-22T21:08:22.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj578 testdata.516403 2026-02-22T21:08:22.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj579 testdata.516403 2026-02-22T21:08:22.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj580 testdata.516403 2026-02-22T21:08:22.461 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj581 testdata.516403 2026-02-22T21:08:22.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj582 testdata.516403 2026-02-22T21:08:22.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj583 testdata.516403 2026-02-22T21:08:22.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj584 testdata.516403 2026-02-22T21:08:22.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj585 testdata.516403 2026-02-22T21:08:22.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj586 testdata.516403 2026-02-22T21:08:22.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj587 testdata.516403 2026-02-22T21:08:22.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj588 testdata.516403 2026-02-22T21:08:22.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj589 testdata.516403 2026-02-22T21:08:22.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj590 testdata.516403 2026-02-22T21:08:22.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj591 testdata.516403 2026-02-22T21:08:22.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj592 testdata.516403 2026-02-22T21:08:22.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj593 testdata.516403 2026-02-22T21:08:22.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj594 testdata.516403 2026-02-22T21:08:22.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj595 testdata.516403 2026-02-22T21:08:22.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj596 testdata.516403 2026-02-22T21:08:22.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj597 testdata.516403 2026-02-22T21:08:22.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj598 testdata.516403 2026-02-22T21:08:22.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj599 testdata.516403 2026-02-22T21:08:22.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj600 testdata.516403 2026-02-22T21:08:22.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj601 testdata.516403 2026-02-22T21:08:22.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj602 testdata.516403 2026-02-22T21:08:22.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.978 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj603 testdata.516403 2026-02-22T21:08:22.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:22.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj604 testdata.516403 2026-02-22T21:08:23.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj605 testdata.516403 2026-02-22T21:08:23.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj606 testdata.516403 2026-02-22T21:08:23.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj607 testdata.516403 2026-02-22T21:08:23.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj608 testdata.516403 2026-02-22T21:08:23.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj609 testdata.516403 2026-02-22T21:08:23.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj610 testdata.516403 2026-02-22T21:08:23.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj611 testdata.516403 2026-02-22T21:08:23.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj612 testdata.516403 2026-02-22T21:08:23.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj613 testdata.516403 2026-02-22T21:08:23.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj614 testdata.516403 2026-02-22T21:08:23.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj615 testdata.516403 2026-02-22T21:08:23.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj616 testdata.516403 2026-02-22T21:08:23.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj617 testdata.516403 2026-02-22T21:08:23.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj618 testdata.516403 2026-02-22T21:08:23.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj619 testdata.516403 2026-02-22T21:08:23.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.362 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj620 testdata.516403 2026-02-22T21:08:23.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj621 testdata.516403 2026-02-22T21:08:23.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj622 testdata.516403 2026-02-22T21:08:23.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj623 testdata.516403 2026-02-22T21:08:23.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj624 testdata.516403 2026-02-22T21:08:23.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj625 testdata.516403 2026-02-22T21:08:23.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj626 testdata.516403 2026-02-22T21:08:23.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj627 testdata.516403 2026-02-22T21:08:23.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj628 testdata.516403 2026-02-22T21:08:23.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj629 testdata.516403 2026-02-22T21:08:23.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj630 testdata.516403 2026-02-22T21:08:23.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.624 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj631 testdata.516403 2026-02-22T21:08:23.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj632 testdata.516403 2026-02-22T21:08:23.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj633 testdata.516403 2026-02-22T21:08:23.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.693 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj634 testdata.516403 2026-02-22T21:08:23.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj635 testdata.516403 2026-02-22T21:08:23.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj636 testdata.516403 2026-02-22T21:08:23.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj637 testdata.516403 2026-02-22T21:08:23.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj638 testdata.516403 2026-02-22T21:08:23.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj639 testdata.516403 2026-02-22T21:08:23.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj640 testdata.516403 2026-02-22T21:08:23.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj641 testdata.516403 2026-02-22T21:08:23.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj642 testdata.516403 2026-02-22T21:08:23.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj643 testdata.516403 2026-02-22T21:08:23.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj644 testdata.516403 2026-02-22T21:08:23.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj645 testdata.516403 2026-02-22T21:08:23.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj646 testdata.516403 2026-02-22T21:08:23.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:23.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj647 testdata.516403 2026-02-22T21:08:24.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj648 testdata.516403 2026-02-22T21:08:24.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj649 testdata.516403 2026-02-22T21:08:24.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj650 testdata.516403 2026-02-22T21:08:24.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj651 testdata.516403 2026-02-22T21:08:24.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj652 testdata.516403 2026-02-22T21:08:24.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj653 testdata.516403 2026-02-22T21:08:24.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj654 testdata.516403 2026-02-22T21:08:24.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj655 testdata.516403 2026-02-22T21:08:24.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj656 testdata.516403 2026-02-22T21:08:24.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj657 testdata.516403 2026-02-22T21:08:24.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj658 testdata.516403 2026-02-22T21:08:24.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj659 testdata.516403 2026-02-22T21:08:24.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj660 testdata.516403 2026-02-22T21:08:24.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.304 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj661 testdata.516403 2026-02-22T21:08:24.321 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj662 testdata.516403 2026-02-22T21:08:24.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj663 testdata.516403 2026-02-22T21:08:24.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj664 testdata.516403 2026-02-22T21:08:24.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj665 testdata.516403 2026-02-22T21:08:24.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj666 testdata.516403 2026-02-22T21:08:24.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj667 testdata.516403 2026-02-22T21:08:24.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj668 testdata.516403 2026-02-22T21:08:24.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj669 testdata.516403 2026-02-22T21:08:24.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj670 testdata.516403 2026-02-22T21:08:24.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj671 testdata.516403 2026-02-22T21:08:24.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj672 testdata.516403 2026-02-22T21:08:24.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj673 testdata.516403 2026-02-22T21:08:24.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj674 testdata.516403 2026-02-22T21:08:24.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj675 testdata.516403 2026-02-22T21:08:24.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj676 testdata.516403 2026-02-22T21:08:24.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj677 testdata.516403 2026-02-22T21:08:24.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj678 testdata.516403 2026-02-22T21:08:24.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj679 testdata.516403 2026-02-22T21:08:24.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj680 testdata.516403 2026-02-22T21:08:24.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj681 testdata.516403 2026-02-22T21:08:24.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj682 testdata.516403 2026-02-22T21:08:24.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj683 testdata.516403 2026-02-22T21:08:24.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj684 testdata.516403 2026-02-22T21:08:24.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj685 testdata.516403 2026-02-22T21:08:24.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj686 testdata.516403 2026-02-22T21:08:24.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj687 testdata.516403 2026-02-22T21:08:24.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj688 testdata.516403 2026-02-22T21:08:24.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj689 testdata.516403 2026-02-22T21:08:24.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:24.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj690 testdata.516403 2026-02-22T21:08:25.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj691 testdata.516403 2026-02-22T21:08:25.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj692 testdata.516403 2026-02-22T21:08:25.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj693 testdata.516403 2026-02-22T21:08:25.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj694 testdata.516403 2026-02-22T21:08:25.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj695 testdata.516403 2026-02-22T21:08:25.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj696 testdata.516403 2026-02-22T21:08:25.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj697 testdata.516403 2026-02-22T21:08:25.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.167 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj698 testdata.516403 2026-02-22T21:08:25.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj699 testdata.516403 2026-02-22T21:08:25.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj700 testdata.516403 2026-02-22T21:08:25.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj701 testdata.516403 2026-02-22T21:08:25.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj702 testdata.516403 2026-02-22T21:08:25.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj703 testdata.516403 2026-02-22T21:08:25.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj704 testdata.516403 2026-02-22T21:08:25.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj705 testdata.516403 2026-02-22T21:08:25.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj706 testdata.516403 2026-02-22T21:08:25.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj707 testdata.516403 2026-02-22T21:08:25.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj708 testdata.516403 2026-02-22T21:08:25.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj709 testdata.516403 2026-02-22T21:08:25.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj710 testdata.516403 2026-02-22T21:08:25.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj711 testdata.516403 2026-02-22T21:08:25.503 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.504 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj712 testdata.516403 2026-02-22T21:08:25.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj713 testdata.516403 2026-02-22T21:08:25.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj714 testdata.516403 2026-02-22T21:08:25.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj715 testdata.516403 2026-02-22T21:08:25.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj716 testdata.516403 2026-02-22T21:08:25.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj717 testdata.516403 2026-02-22T21:08:25.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj718 testdata.516403 2026-02-22T21:08:25.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj719 testdata.516403 2026-02-22T21:08:25.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj720 testdata.516403 2026-02-22T21:08:25.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj721 testdata.516403 2026-02-22T21:08:25.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj722 testdata.516403 2026-02-22T21:08:25.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.773 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj723 testdata.516403 2026-02-22T21:08:25.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj724 testdata.516403 2026-02-22T21:08:25.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj725 testdata.516403 2026-02-22T21:08:25.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj726 testdata.516403 2026-02-22T21:08:25.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj727 testdata.516403 2026-02-22T21:08:25.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj728 testdata.516403 2026-02-22T21:08:25.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj729 testdata.516403 2026-02-22T21:08:25.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.925 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj730 testdata.516403 2026-02-22T21:08:25.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj731 testdata.516403 2026-02-22T21:08:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj732 testdata.516403 2026-02-22T21:08:25.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:25.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj733 testdata.516403 2026-02-22T21:08:26.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj734 testdata.516403 2026-02-22T21:08:26.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj735 testdata.516403 2026-02-22T21:08:26.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj736 testdata.516403 2026-02-22T21:08:26.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj737 testdata.516403 2026-02-22T21:08:26.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj738 testdata.516403 2026-02-22T21:08:26.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj739 testdata.516403 2026-02-22T21:08:26.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj740 testdata.516403 2026-02-22T21:08:26.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj741 testdata.516403 2026-02-22T21:08:26.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj742 testdata.516403 2026-02-22T21:08:26.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj743 testdata.516403 2026-02-22T21:08:26.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj744 testdata.516403 2026-02-22T21:08:26.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj745 testdata.516403 2026-02-22T21:08:26.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj746 testdata.516403 2026-02-22T21:08:26.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj747 testdata.516403 2026-02-22T21:08:26.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj748 testdata.516403 2026-02-22T21:08:26.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj749 testdata.516403 2026-02-22T21:08:26.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj750 testdata.516403 2026-02-22T21:08:26.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj751 testdata.516403 2026-02-22T21:08:26.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj752 testdata.516403 2026-02-22T21:08:26.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj753 testdata.516403 2026-02-22T21:08:26.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj754 testdata.516403 2026-02-22T21:08:26.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj755 testdata.516403 2026-02-22T21:08:26.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj756 testdata.516403 2026-02-22T21:08:26.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj757 testdata.516403 2026-02-22T21:08:26.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj758 testdata.516403 2026-02-22T21:08:26.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj759 testdata.516403 2026-02-22T21:08:26.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj760 testdata.516403 2026-02-22T21:08:26.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj761 testdata.516403 2026-02-22T21:08:26.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj762 testdata.516403 2026-02-22T21:08:26.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj763 testdata.516403 2026-02-22T21:08:26.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj764 testdata.516403 2026-02-22T21:08:26.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj765 testdata.516403 2026-02-22T21:08:26.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj766 testdata.516403 2026-02-22T21:08:26.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj767 testdata.516403 2026-02-22T21:08:26.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj768 testdata.516403 2026-02-22T21:08:26.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj769 testdata.516403 2026-02-22T21:08:26.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.863 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj770 testdata.516403 2026-02-22T21:08:26.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj771 testdata.516403 2026-02-22T21:08:26.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj772 testdata.516403 2026-02-22T21:08:26.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj773 testdata.516403 2026-02-22T21:08:26.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj774 testdata.516403 2026-02-22T21:08:26.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj775 testdata.516403 2026-02-22T21:08:26.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:26.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj776 testdata.516403 2026-02-22T21:08:27.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj777 testdata.516403 2026-02-22T21:08:27.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj778 testdata.516403 2026-02-22T21:08:27.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj779 testdata.516403 2026-02-22T21:08:27.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj780 testdata.516403 2026-02-22T21:08:27.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj781 testdata.516403 2026-02-22T21:08:27.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj782 testdata.516403 2026-02-22T21:08:27.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.140 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj783 testdata.516403 2026-02-22T21:08:27.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.165 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj784 testdata.516403 2026-02-22T21:08:27.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj785 testdata.516403 2026-02-22T21:08:27.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj786 testdata.516403 2026-02-22T21:08:27.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj787 testdata.516403 2026-02-22T21:08:27.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj788 testdata.516403 2026-02-22T21:08:27.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj789 testdata.516403 2026-02-22T21:08:27.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj790 testdata.516403 2026-02-22T21:08:27.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.303 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj791 testdata.516403 2026-02-22T21:08:27.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.327 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj792 testdata.516403 2026-02-22T21:08:27.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj793 testdata.516403 2026-02-22T21:08:27.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj794 testdata.516403 2026-02-22T21:08:27.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj795 testdata.516403 2026-02-22T21:08:27.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj796 testdata.516403 2026-02-22T21:08:27.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj797 testdata.516403 2026-02-22T21:08:27.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.471 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj798 testdata.516403 2026-02-22T21:08:27.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj799 testdata.516403 2026-02-22T21:08:27.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj800 testdata.516403 2026-02-22T21:08:27.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj801 testdata.516403 2026-02-22T21:08:27.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj802 testdata.516403 2026-02-22T21:08:27.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj803 testdata.516403 2026-02-22T21:08:27.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj804 testdata.516403 2026-02-22T21:08:27.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.625 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj805 testdata.516403 2026-02-22T21:08:27.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj806 testdata.516403 2026-02-22T21:08:27.674 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.675 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj807 testdata.516403 2026-02-22T21:08:27.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj808 testdata.516403 2026-02-22T21:08:27.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj809 testdata.516403 2026-02-22T21:08:27.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj810 testdata.516403 2026-02-22T21:08:27.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj811 testdata.516403 2026-02-22T21:08:27.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj812 testdata.516403 2026-02-22T21:08:27.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj813 testdata.516403 2026-02-22T21:08:27.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj814 testdata.516403 2026-02-22T21:08:27.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj815 testdata.516403 2026-02-22T21:08:27.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj816 testdata.516403 2026-02-22T21:08:27.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj817 testdata.516403 2026-02-22T21:08:27.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj818 testdata.516403 2026-02-22T21:08:27.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj819 testdata.516403 2026-02-22T21:08:27.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:27.988 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj820 testdata.516403 2026-02-22T21:08:28.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj821 testdata.516403 2026-02-22T21:08:28.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj822 testdata.516403 2026-02-22T21:08:28.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj823 testdata.516403 2026-02-22T21:08:28.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj824 testdata.516403 2026-02-22T21:08:28.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj825 testdata.516403 2026-02-22T21:08:28.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj826 testdata.516403 2026-02-22T21:08:28.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj827 testdata.516403 2026-02-22T21:08:28.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj828 testdata.516403 2026-02-22T21:08:28.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj829 testdata.516403 2026-02-22T21:08:28.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj830 testdata.516403 2026-02-22T21:08:28.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj831 testdata.516403 2026-02-22T21:08:28.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj832 testdata.516403 2026-02-22T21:08:28.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj833 testdata.516403 2026-02-22T21:08:28.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj834 testdata.516403 2026-02-22T21:08:28.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj835 testdata.516403 2026-02-22T21:08:28.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj836 testdata.516403 2026-02-22T21:08:28.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj837 testdata.516403 2026-02-22T21:08:28.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj838 testdata.516403 2026-02-22T21:08:28.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.456 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj839 testdata.516403 2026-02-22T21:08:28.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.481 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj840 testdata.516403 2026-02-22T21:08:28.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.505 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj841 testdata.516403 2026-02-22T21:08:28.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj842 testdata.516403 2026-02-22T21:08:28.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj843 testdata.516403 2026-02-22T21:08:28.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.574 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj844 testdata.516403 2026-02-22T21:08:28.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj845 testdata.516403 2026-02-22T21:08:28.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj846 testdata.516403 2026-02-22T21:08:28.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj847 testdata.516403 2026-02-22T21:08:28.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj848 testdata.516403 2026-02-22T21:08:28.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj849 testdata.516403 2026-02-22T21:08:28.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj850 testdata.516403 2026-02-22T21:08:28.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj851 testdata.516403 2026-02-22T21:08:28.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj852 testdata.516403 2026-02-22T21:08:28.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj853 testdata.516403 2026-02-22T21:08:28.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj854 testdata.516403 2026-02-22T21:08:28.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj855 testdata.516403 2026-02-22T21:08:28.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj856 testdata.516403 2026-02-22T21:08:28.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj857 testdata.516403 2026-02-22T21:08:28.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj858 testdata.516403 2026-02-22T21:08:28.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj859 testdata.516403 2026-02-22T21:08:28.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj860 testdata.516403 2026-02-22T21:08:28.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.951 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj861 testdata.516403 2026-02-22T21:08:28.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:28.976 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj862 testdata.516403 2026-02-22T21:08:29.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj863 testdata.516403 2026-02-22T21:08:29.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj864 testdata.516403 2026-02-22T21:08:29.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj865 testdata.516403 2026-02-22T21:08:29.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj866 testdata.516403 2026-02-22T21:08:29.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj867 testdata.516403 2026-02-22T21:08:29.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj868 testdata.516403 2026-02-22T21:08:29.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj869 testdata.516403 2026-02-22T21:08:29.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj870 testdata.516403 2026-02-22T21:08:29.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj871 testdata.516403 2026-02-22T21:08:29.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj872 testdata.516403 2026-02-22T21:08:29.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj873 testdata.516403 2026-02-22T21:08:29.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj874 testdata.516403 2026-02-22T21:08:29.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.285 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj875 testdata.516403 2026-02-22T21:08:29.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj876 testdata.516403 2026-02-22T21:08:29.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj877 testdata.516403 2026-02-22T21:08:29.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.360 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj878 testdata.516403 2026-02-22T21:08:29.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj879 testdata.516403 2026-02-22T21:08:29.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj880 testdata.516403 2026-02-22T21:08:29.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj881 testdata.516403 2026-02-22T21:08:29.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj882 testdata.516403 2026-02-22T21:08:29.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj883 testdata.516403 2026-02-22T21:08:29.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj884 testdata.516403 2026-02-22T21:08:29.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.532 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj885 testdata.516403 2026-02-22T21:08:29.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj886 testdata.516403 2026-02-22T21:08:29.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.582 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj887 testdata.516403 2026-02-22T21:08:29.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj888 testdata.516403 2026-02-22T21:08:29.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj889 testdata.516403 2026-02-22T21:08:29.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj890 testdata.516403 2026-02-22T21:08:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj891 testdata.516403 2026-02-22T21:08:29.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj892 testdata.516403 2026-02-22T21:08:29.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj893 testdata.516403 2026-02-22T21:08:29.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj894 testdata.516403 2026-02-22T21:08:29.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj895 testdata.516403 2026-02-22T21:08:29.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj896 testdata.516403 2026-02-22T21:08:29.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj897 testdata.516403 2026-02-22T21:08:29.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj898 testdata.516403 2026-02-22T21:08:29.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj899 testdata.516403 2026-02-22T21:08:29.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj900 testdata.516403 2026-02-22T21:08:29.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj901 testdata.516403 2026-02-22T21:08:29.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj902 testdata.516403 2026-02-22T21:08:29.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj903 testdata.516403 2026-02-22T21:08:29.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:29.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj904 testdata.516403 2026-02-22T21:08:30.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj905 testdata.516403 2026-02-22T21:08:30.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj906 testdata.516403 2026-02-22T21:08:30.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj907 testdata.516403 2026-02-22T21:08:30.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj908 testdata.516403 2026-02-22T21:08:30.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.102 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj909 testdata.516403 2026-02-22T21:08:30.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj910 testdata.516403 2026-02-22T21:08:30.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj911 testdata.516403 2026-02-22T21:08:30.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj912 testdata.516403 2026-02-22T21:08:30.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.200 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj913 testdata.516403 2026-02-22T21:08:30.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj914 testdata.516403 2026-02-22T21:08:30.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj915 testdata.516403 2026-02-22T21:08:30.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.274 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj916 testdata.516403 2026-02-22T21:08:30.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj917 testdata.516403 2026-02-22T21:08:30.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj918 testdata.516403 2026-02-22T21:08:30.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj919 testdata.516403 2026-02-22T21:08:30.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj920 testdata.516403 2026-02-22T21:08:30.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj921 testdata.516403 2026-02-22T21:08:30.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj922 testdata.516403 2026-02-22T21:08:30.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj923 testdata.516403 2026-02-22T21:08:30.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj924 testdata.516403 2026-02-22T21:08:30.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj925 testdata.516403 2026-02-22T21:08:30.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj926 testdata.516403 2026-02-22T21:08:30.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj927 testdata.516403 2026-02-22T21:08:30.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj928 testdata.516403 2026-02-22T21:08:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj929 testdata.516403 2026-02-22T21:08:30.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj930 testdata.516403 2026-02-22T21:08:30.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj931 testdata.516403 2026-02-22T21:08:30.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj932 testdata.516403 2026-02-22T21:08:30.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj933 testdata.516403 2026-02-22T21:08:30.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj934 testdata.516403 2026-02-22T21:08:30.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj935 testdata.516403 2026-02-22T21:08:30.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj936 testdata.516403 2026-02-22T21:08:30.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj937 testdata.516403 2026-02-22T21:08:30.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj938 testdata.516403 2026-02-22T21:08:30.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj939 testdata.516403 2026-02-22T21:08:30.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj940 testdata.516403 2026-02-22T21:08:30.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj941 testdata.516403 2026-02-22T21:08:30.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj942 testdata.516403 2026-02-22T21:08:30.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj943 testdata.516403 2026-02-22T21:08:30.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj944 testdata.516403 2026-02-22T21:08:30.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.962 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj945 testdata.516403 2026-02-22T21:08:30.983 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:30.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj946 testdata.516403 2026-02-22T21:08:31.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj947 testdata.516403 2026-02-22T21:08:31.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj948 testdata.516403 2026-02-22T21:08:31.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj949 testdata.516403 2026-02-22T21:08:31.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj950 testdata.516403 2026-02-22T21:08:31.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj951 testdata.516403 2026-02-22T21:08:31.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj952 testdata.516403 2026-02-22T21:08:31.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj953 testdata.516403 2026-02-22T21:08:31.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj954 testdata.516403 2026-02-22T21:08:31.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj955 testdata.516403 2026-02-22T21:08:31.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.218 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj956 testdata.516403 2026-02-22T21:08:31.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj957 testdata.516403 2026-02-22T21:08:31.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj958 testdata.516403 2026-02-22T21:08:31.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj959 testdata.516403 2026-02-22T21:08:31.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj960 testdata.516403 2026-02-22T21:08:31.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj961 testdata.516403 2026-02-22T21:08:31.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.358 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj962 testdata.516403 2026-02-22T21:08:31.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj963 testdata.516403 2026-02-22T21:08:31.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj964 testdata.516403 2026-02-22T21:08:31.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj965 testdata.516403 2026-02-22T21:08:31.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj966 testdata.516403 2026-02-22T21:08:31.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj967 testdata.516403 2026-02-22T21:08:31.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj968 testdata.516403 2026-02-22T21:08:31.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj969 testdata.516403 2026-02-22T21:08:31.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.541 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj970 testdata.516403 2026-02-22T21:08:31.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj971 testdata.516403 2026-02-22T21:08:31.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj972 testdata.516403 2026-02-22T21:08:31.600 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj973 testdata.516403 2026-02-22T21:08:31.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj974 testdata.516403 2026-02-22T21:08:31.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.647 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj975 testdata.516403 2026-02-22T21:08:31.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj976 testdata.516403 2026-02-22T21:08:31.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj977 testdata.516403 2026-02-22T21:08:31.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj978 testdata.516403 2026-02-22T21:08:31.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj979 testdata.516403 2026-02-22T21:08:31.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj980 testdata.516403 2026-02-22T21:08:31.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj981 testdata.516403 2026-02-22T21:08:31.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.821 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj982 testdata.516403 2026-02-22T21:08:31.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj983 testdata.516403 2026-02-22T21:08:31.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj984 testdata.516403 2026-02-22T21:08:31.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj985 testdata.516403 2026-02-22T21:08:31.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj986 testdata.516403 2026-02-22T21:08:31.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj987 testdata.516403 2026-02-22T21:08:31.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj988 testdata.516403 2026-02-22T21:08:31.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:31.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj989 testdata.516403 2026-02-22T21:08:32.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:32.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj990 testdata.516403 2026-02-22T21:08:32.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:32.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj991 testdata.516403 2026-02-22T21:08:32.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:32.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj992 testdata.516403 2026-02-22T21:08:32.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:32.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj993 testdata.516403 2026-02-22T21:08:32.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:32.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj994 testdata.516403 2026-02-22T21:08:32.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:32.141 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj995 testdata.516403 2026-02-22T21:08:32.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:32.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj996 testdata.516403 2026-02-22T21:08:32.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:32.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj997 testdata.516403 2026-02-22T21:08:32.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:32.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj998 testdata.516403 2026-02-22T21:08:32.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:32.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj999 testdata.516403 2026-02-22T21:08:32.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:330: _scrub_abort: for i in `seq 1 $objects` 2026-02-22T21:08:32.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:332: _scrub_abort: rados -p test put obj1000 testdata.516403 2026-02-22T21:08:32.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:334: _scrub_abort: rm -f testdata.516403 2026-02-22T21:08:32.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: get_primary test obj1 2026-02-22T21:08:32.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-22T21:08:32.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T21:08:32.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-22T21:08:32.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T21:08:32.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:336: _scrub_abort: local primary=1 2026-02-22T21:08:32.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:337: _scrub_abort: local pgid=1.0 2026-02-22T21:08:32.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:339: _scrub_abort: ceph tell 1.0 schedule-scrub 2026-02-22T21:08:32.518 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:08:32.518 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T21:08:32.518 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T21:08:32.518 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-20T21:08:32.513200+0000" 2026-02-22T21:08:32.518 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:08:32.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:342: _scrub_abort: set -o pipefail 2026-02-22T21:08:32.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:343: _scrub_abort: found=no 2026-02-22T21:08:32.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: seq 0 200 2026-02-22T21:08:32.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:32.526 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: flush_pg_stats 2026-02-22T21:08:32.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:32.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:32.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:32.684 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:32.685 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:32.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:32.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:32.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:32.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836515 2026-02-22T21:08:32.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836515 2026-02-22T21:08:32.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515' 2026-02-22T21:08:32.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:32.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:32.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705696 2026-02-22T21:08:32.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705696 2026-02-22T21:08:32.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-38654705696' 2026-02-22T21:08:32.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:32.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:32.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542174 2026-02-22T21:08:32.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542174 2026-02-22T21:08:32.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836515 1-38654705696 2-60129542174' 2026-02-22T21:08:32.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:32.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836515 2026-02-22T21:08:32.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:32.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:32.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836515 2026-02-22T21:08:32.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:32.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836515 2026-02-22T21:08:32.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2026-02-22T21:08:32.903 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836515 2026-02-22T21:08:32.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:33.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836513 -lt 21474836515 2026-02-22T21:08:33.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:34.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:08:34.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:34.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836515 2026-02-22T21:08:34.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:34.229 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705696 2026-02-22T21:08:34.229 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:34.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:34.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705696 2026-02-22T21:08:34.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:34.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705696 2026-02-22T21:08:34.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705696' 2026-02-22T21:08:34.233 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705696 2026-02-22T21:08:34.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:34.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705697 -lt 38654705696 2026-02-22T21:08:34.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:34.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542174 2026-02-22T21:08:34.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:34.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:34.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542174 2026-02-22T21:08:34.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:34.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542174 2026-02-22T21:08:34.399 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542174 2026-02-22T21:08:34.400 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542174' 2026-02-22T21:08:34.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:34.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542175 -lt 60129542174 2026-02-22T21:08:34.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: ceph pg dump pgs 2026-02-22T21:08:34.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep '^1.0' 2026-02-22T21:08:34.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep -q scrubbing 2026-02-22T21:08:34.717 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:08:34.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:34.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: flush_pg_stats 2026-02-22T21:08:34.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:34.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:34.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:34.881 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:34.881 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:34.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:34.881 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:34.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:34.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836518 2026-02-22T21:08:34.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836518 2026-02-22T21:08:34.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518' 2026-02-22T21:08:34.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:34.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:35.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705700 2026-02-22T21:08:35.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705700 2026-02-22T21:08:35.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705700' 2026-02-22T21:08:35.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:35.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:35.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542177 2026-02-22T21:08:35.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542177 2026-02-22T21:08:35.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836518 1-38654705700 2-60129542177' 2026-02-22T21:08:35.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:35.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836518 2026-02-22T21:08:35.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:35.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:35.100 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836518 2026-02-22T21:08:35.100 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:35.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836518 2026-02-22T21:08:35.101 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836518 2026-02-22T21:08:35.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2026-02-22T21:08:35.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:35.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836516 -lt 21474836518 2026-02-22T21:08:35.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:36.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:08:36.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:36.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836519 -lt 21474836518 2026-02-22T21:08:36.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:36.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705700 2026-02-22T21:08:36.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:36.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:36.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705700 2026-02-22T21:08:36.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:36.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705700 2026-02-22T21:08:36.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705700' 2026-02-22T21:08:36.423 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705700 2026-02-22T21:08:36.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705700 -lt 38654705700 2026-02-22T21:08:36.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:36.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542177 2026-02-22T21:08:36.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:36.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:36.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542177 2026-02-22T21:08:36.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:36.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542177 2026-02-22T21:08:36.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542177' 2026-02-22T21:08:36.595 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542177 2026-02-22T21:08:36.596 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:36.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542178 -lt 60129542177 2026-02-22T21:08:36.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: ceph pg dump pgs 2026-02-22T21:08:36.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep '^1.0' 2026-02-22T21:08:36.755 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep -q scrubbing 2026-02-22T21:08:36.900 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:08:36.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:344: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:36.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:346: _scrub_abort: flush_pg_stats 2026-02-22T21:08:36.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:36.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:37.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:37.074 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:37.074 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:37.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:37.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:37.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:37.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836521 2026-02-22T21:08:37.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836521 2026-02-22T21:08:37.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836521' 2026-02-22T21:08:37.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:37.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:37.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705703 2026-02-22T21:08:37.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705703 2026-02-22T21:08:37.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836521 1-38654705703' 2026-02-22T21:08:37.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:37.219 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:37.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542180 2026-02-22T21:08:37.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542180 2026-02-22T21:08:37.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836521 1-38654705703 2-60129542180' 2026-02-22T21:08:37.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:37.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836521 2026-02-22T21:08:37.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:37.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:37.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836521 2026-02-22T21:08:37.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:37.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836521 2026-02-22T21:08:37.297 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836521 2026-02-22T21:08:37.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2026-02-22T21:08:37.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:37.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836519 -lt 21474836521 2026-02-22T21:08:37.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:38.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:08:38.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:38.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836522 -lt 21474836521 2026-02-22T21:08:38.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:38.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705703 2026-02-22T21:08:38.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:38.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:38.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705703 2026-02-22T21:08:38.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:38.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705703 2026-02-22T21:08:38.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705703' 2026-02-22T21:08:38.618 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705703 2026-02-22T21:08:38.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:38.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705703 -lt 38654705703 2026-02-22T21:08:38.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:38.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542180 2026-02-22T21:08:38.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:38.776 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:38.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542180 2026-02-22T21:08:38.777 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:38.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542180 2026-02-22T21:08:38.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542180' 2026-02-22T21:08:38.779 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542180 2026-02-22T21:08:38.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:38.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542181 -lt 60129542180 2026-02-22T21:08:38.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: ceph pg dump pgs 2026-02-22T21:08:38.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep '^1.0' 2026-02-22T21:08:38.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:347: _scrub_abort: grep -q scrubbing 2026-02-22T21:08:39.096 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:08:39.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:349: _scrub_abort: found=yes 2026-02-22T21:08:39.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:351: _scrub_abort: break 2026-02-22T21:08:39.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:354: _scrub_abort: set +o pipefail 2026-02-22T21:08:39.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:356: _scrub_abort: test yes = no 2026-02-22T21:08:39.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:362: _scrub_abort: ceph osd set noscrub 2026-02-22T21:08:39.304 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is set 2026-02-22T21:08:39.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:363: _scrub_abort: '[' scrub = deep-scrub ']' 2026-02-22T21:08:39.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:369: _scrub_abort: set -o pipefail 2026-02-22T21:08:39.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: seq 0 200 2026-02-22T21:08:39.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:39.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:08:39.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:39.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:39.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:39.476 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:39.476 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:39.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:39.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:39.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:39.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836525 2026-02-22T21:08:39.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836525 2026-02-22T21:08:39.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836525' 2026-02-22T21:08:39.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:39.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:39.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705706 2026-02-22T21:08:39.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705706 2026-02-22T21:08:39.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836525 1-38654705706' 2026-02-22T21:08:39.623 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:39.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:39.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542184 2026-02-22T21:08:39.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542184 2026-02-22T21:08:39.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836525 1-38654705706 2-60129542184' 2026-02-22T21:08:39.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:39.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836525 2026-02-22T21:08:39.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:39.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:39.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836525 2026-02-22T21:08:39.697 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:39.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836525 2026-02-22T21:08:39.699 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836525 2026-02-22T21:08:39.699 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2026-02-22T21:08:39.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:39.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836524 -lt 21474836525 2026-02-22T21:08:39.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:40.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:08:40.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:41.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836524 -lt 21474836525 2026-02-22T21:08:41.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:42.028 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-22T21:08:42.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:42.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836527 -lt 21474836525 2026-02-22T21:08:42.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:42.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705706 2026-02-22T21:08:42.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:42.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:42.189 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705706 2026-02-22T21:08:42.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:42.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705706 2026-02-22T21:08:42.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705706' 2026-02-22T21:08:42.191 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705706 2026-02-22T21:08:42.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:42.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705708 -lt 38654705706 2026-02-22T21:08:42.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:42.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542184 2026-02-22T21:08:42.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:42.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:42.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542184 2026-02-22T21:08:42.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:42.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542184 2026-02-22T21:08:42.424 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542184 2026-02-22T21:08:42.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542184' 2026-02-22T21:08:42.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:42.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542186 -lt 60129542184 2026-02-22T21:08:42.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:08:42.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:08:42.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:08:42.738 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:08:42.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:08:42.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:42.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:08:42.748 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:42.749 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:42.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:42.907 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:42.907 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:42.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:42.907 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:42.908 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:42.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836529 2026-02-22T21:08:42.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836529 2026-02-22T21:08:42.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836529' 2026-02-22T21:08:42.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:42.981 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:43.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705711 2026-02-22T21:08:43.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705711 2026-02-22T21:08:43.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836529 1-38654705711' 2026-02-22T21:08:43.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:43.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:43.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542188 2026-02-22T21:08:43.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542188 2026-02-22T21:08:43.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836529 1-38654705711 2-60129542188' 2026-02-22T21:08:43.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:43.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836529 2026-02-22T21:08:43.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:43.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:43.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836529 2026-02-22T21:08:43.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:43.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836529 2026-02-22T21:08:43.126 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836529 2026-02-22T21:08:43.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2026-02-22T21:08:43.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:43.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836527 -lt 21474836529 2026-02-22T21:08:43.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:44.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:08:44.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:44.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836530 -lt 21474836529 2026-02-22T21:08:44.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:44.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705711 2026-02-22T21:08:44.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:44.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:44.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705711 2026-02-22T21:08:44.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:44.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705711 2026-02-22T21:08:44.449 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705711 2026-02-22T21:08:44.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705711' 2026-02-22T21:08:44.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:44.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705711 -lt 38654705711 2026-02-22T21:08:44.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:44.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542188 2026-02-22T21:08:44.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:44.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:44.613 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542188 2026-02-22T21:08:44.613 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:44.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542188 2026-02-22T21:08:44.615 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542188 2026-02-22T21:08:44.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542188' 2026-02-22T21:08:44.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:44.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542189 -lt 60129542188 2026-02-22T21:08:44.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:08:44.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:08:44.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:08:44.928 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:08:44.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:08:44.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:44.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:08:44.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:44.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:45.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:45.093 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:45.093 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:45.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:45.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:45.094 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:45.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836532 2026-02-22T21:08:45.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836532 2026-02-22T21:08:45.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532' 2026-02-22T21:08:45.166 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:45.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:45.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705714 2026-02-22T21:08:45.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705714 2026-02-22T21:08:45.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532 1-38654705714' 2026-02-22T21:08:45.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:45.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:45.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542191 2026-02-22T21:08:45.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542191 2026-02-22T21:08:45.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532 1-38654705714 2-60129542191' 2026-02-22T21:08:45.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:45.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836532 2026-02-22T21:08:45.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:45.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:45.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836532 2026-02-22T21:08:45.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:45.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836532 2026-02-22T21:08:45.315 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836532 2026-02-22T21:08:45.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2026-02-22T21:08:45.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:45.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836530 -lt 21474836532 2026-02-22T21:08:45.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:46.479 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:08:46.480 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:46.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836533 -lt 21474836532 2026-02-22T21:08:46.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:46.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705714 2026-02-22T21:08:46.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:46.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:46.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705714 2026-02-22T21:08:46.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:46.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705714 2026-02-22T21:08:46.641 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705714 2026-02-22T21:08:46.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705714' 2026-02-22T21:08:46.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:46.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705714 -lt 38654705714 2026-02-22T21:08:46.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:46.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542191 2026-02-22T21:08:46.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:46.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:46.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542191 2026-02-22T21:08:46.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:46.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542191 2026-02-22T21:08:46.806 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542191 2026-02-22T21:08:46.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542191' 2026-02-22T21:08:46.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:46.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542192 -lt 60129542191 2026-02-22T21:08:46.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:08:46.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:08:46.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:08:47.109 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:08:47.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:08:47.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:47.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:08:47.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:47.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:47.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:47.272 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:47.272 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:47.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:47.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:47.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:47.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836535 2026-02-22T21:08:47.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836535 2026-02-22T21:08:47.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836535' 2026-02-22T21:08:47.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:47.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:47.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705717 2026-02-22T21:08:47.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705717 2026-02-22T21:08:47.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836535 1-38654705717' 2026-02-22T21:08:47.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:47.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:47.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542195 2026-02-22T21:08:47.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542195 2026-02-22T21:08:47.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836535 1-38654705717 2-60129542195' 2026-02-22T21:08:47.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:47.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836535 2026-02-22T21:08:47.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:47.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:47.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836535 2026-02-22T21:08:47.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:47.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836535 2026-02-22T21:08:47.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2026-02-22T21:08:47.492 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836535 2026-02-22T21:08:47.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:47.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836536 -lt 21474836535 2026-02-22T21:08:47.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:47.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705717 2026-02-22T21:08:47.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:47.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:47.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705717 2026-02-22T21:08:47.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:47.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705717 2026-02-22T21:08:47.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705717' 2026-02-22T21:08:47.661 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705717 2026-02-22T21:08:47.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:47.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705717 -lt 38654705717 2026-02-22T21:08:47.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:47.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542195 2026-02-22T21:08:47.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:47.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:47.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542195 2026-02-22T21:08:47.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:47.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542195 2026-02-22T21:08:47.828 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542195 2026-02-22T21:08:47.828 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542195' 2026-02-22T21:08:47.828 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:47.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542195 -lt 60129542195 2026-02-22T21:08:47.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:08:47.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:08:47.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:08:48.140 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:08:48.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:08:48.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:48.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:08:48.150 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:48.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:48.306 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:48.307 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:48.307 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:48.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:48.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:48.307 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:48.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836538 2026-02-22T21:08:48.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836538 2026-02-22T21:08:48.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836538' 2026-02-22T21:08:48.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:48.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:48.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705719 2026-02-22T21:08:48.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705719 2026-02-22T21:08:48.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836538 1-38654705719' 2026-02-22T21:08:48.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:48.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:48.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542197 2026-02-22T21:08:48.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542197 2026-02-22T21:08:48.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836538 1-38654705719 2-60129542197' 2026-02-22T21:08:48.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:48.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836538 2026-02-22T21:08:48.524 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:48.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:48.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836538 2026-02-22T21:08:48.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:48.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836538 2026-02-22T21:08:48.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2026-02-22T21:08:48.528 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836538 2026-02-22T21:08:48.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:48.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836536 -lt 21474836538 2026-02-22T21:08:48.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:49.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:08:49.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:49.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836539 -lt 21474836538 2026-02-22T21:08:49.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:49.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705719 2026-02-22T21:08:49.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:49.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:49.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705719 2026-02-22T21:08:49.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:49.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705719 2026-02-22T21:08:49.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705719' 2026-02-22T21:08:49.859 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705719 2026-02-22T21:08:49.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:50.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705720 -lt 38654705719 2026-02-22T21:08:50.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:50.023 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542197 2026-02-22T21:08:50.024 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:50.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:50.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542197 2026-02-22T21:08:50.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:50.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542197 2026-02-22T21:08:50.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542197' 2026-02-22T21:08:50.027 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542197 2026-02-22T21:08:50.027 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:50.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542198 -lt 60129542197 2026-02-22T21:08:50.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:08:50.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:08:50.196 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:08:50.341 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:08:50.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:08:50.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:50.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:08:50.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:50.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:50.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:50.513 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:50.513 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:50.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:50.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:50.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:50.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836541 2026-02-22T21:08:50.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836541 2026-02-22T21:08:50.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836541' 2026-02-22T21:08:50.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:50.586 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:50.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705722 2026-02-22T21:08:50.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705722 2026-02-22T21:08:50.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836541 1-38654705722' 2026-02-22T21:08:50.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:50.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:50.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542200 2026-02-22T21:08:50.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542200 2026-02-22T21:08:50.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836541 1-38654705722 2-60129542200' 2026-02-22T21:08:50.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:50.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836541 2026-02-22T21:08:50.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:50.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:50.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836541 2026-02-22T21:08:50.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:50.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836541 2026-02-22T21:08:50.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2026-02-22T21:08:50.734 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836541 2026-02-22T21:08:50.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:50.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836539 -lt 21474836541 2026-02-22T21:08:50.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:51.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:08:51.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:52.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836542 -lt 21474836541 2026-02-22T21:08:52.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:52.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705722 2026-02-22T21:08:52.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:52.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:52.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705722 2026-02-22T21:08:52.054 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:52.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705722 2026-02-22T21:08:52.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705722' 2026-02-22T21:08:52.056 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705722 2026-02-22T21:08:52.057 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:52.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705723 -lt 38654705722 2026-02-22T21:08:52.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:52.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542200 2026-02-22T21:08:52.226 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:52.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:52.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542200 2026-02-22T21:08:52.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:52.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542200 2026-02-22T21:08:52.230 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542200' 2026-02-22T21:08:52.230 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542200 2026-02-22T21:08:52.230 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:52.390 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542201 -lt 60129542200 2026-02-22T21:08:52.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:08:52.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:08:52.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:08:52.539 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:08:52.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:08:52.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:52.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:08:52.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:52.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:52.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:52.704 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:52.704 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:52.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:52.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:52.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:52.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836544 2026-02-22T21:08:52.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836544 2026-02-22T21:08:52.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836544' 2026-02-22T21:08:52.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:52.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:52.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705725 2026-02-22T21:08:52.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705725 2026-02-22T21:08:52.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836544 1-38654705725' 2026-02-22T21:08:52.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:52.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542203 2026-02-22T21:08:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542203 2026-02-22T21:08:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836544 1-38654705725 2-60129542203' 2026-02-22T21:08:52.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:52.923 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836544 2026-02-22T21:08:52.923 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:52.924 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:52.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836544 2026-02-22T21:08:52.925 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:52.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836544 2026-02-22T21:08:52.927 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836544 2026-02-22T21:08:52.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2026-02-22T21:08:52.927 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:53.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836542 -lt 21474836544 2026-02-22T21:08:53.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:54.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:08:54.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:54.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836545 -lt 21474836544 2026-02-22T21:08:54.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:54.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705725 2026-02-22T21:08:54.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:54.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:54.254 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705725 2026-02-22T21:08:54.254 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:54.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705725 2026-02-22T21:08:54.256 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705725 2026-02-22T21:08:54.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705725' 2026-02-22T21:08:54.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:54.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705726 -lt 38654705725 2026-02-22T21:08:54.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:54.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542203 2026-02-22T21:08:54.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:54.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:54.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542203 2026-02-22T21:08:54.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:54.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542203 2026-02-22T21:08:54.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542203' 2026-02-22T21:08:54.423 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542203 2026-02-22T21:08:54.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:54.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542204 -lt 60129542203 2026-02-22T21:08:54.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:08:54.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:08:54.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:08:54.730 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:08:54.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:08:54.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:54.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:08:54.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:54.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:54.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:54.902 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:54.902 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:54.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:54.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:54.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:54.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836547 2026-02-22T21:08:54.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836547 2026-02-22T21:08:54.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836547' 2026-02-22T21:08:54.975 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:54.975 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:55.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705729 2026-02-22T21:08:55.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705729 2026-02-22T21:08:55.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836547 1-38654705729' 2026-02-22T21:08:55.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:55.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:55.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542206 2026-02-22T21:08:55.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542206 2026-02-22T21:08:55.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836547 1-38654705729 2-60129542206' 2026-02-22T21:08:55.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:55.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836547 2026-02-22T21:08:55.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:55.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:55.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836547 2026-02-22T21:08:55.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:55.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836547 2026-02-22T21:08:55.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836547' 2026-02-22T21:08:55.123 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836547 2026-02-22T21:08:55.123 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:55.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836545 -lt 21474836547 2026-02-22T21:08:55.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:56.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:08:56.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:56.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836548 -lt 21474836547 2026-02-22T21:08:56.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:56.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705729 2026-02-22T21:08:56.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:56.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:56.455 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705729 2026-02-22T21:08:56.455 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:56.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705729 2026-02-22T21:08:56.457 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705729' 2026-02-22T21:08:56.457 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705729 2026-02-22T21:08:56.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:56.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705729 -lt 38654705729 2026-02-22T21:08:56.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:56.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542206 2026-02-22T21:08:56.617 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:56.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:56.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542206 2026-02-22T21:08:56.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:56.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542206 2026-02-22T21:08:56.621 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542206 2026-02-22T21:08:56.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542206' 2026-02-22T21:08:56.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:56.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542207 -lt 60129542206 2026-02-22T21:08:56.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:08:56.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:08:56.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:08:56.927 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:08:56.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:08:56.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:56.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:08:56.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:56.937 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:57.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:57.101 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:57.101 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:57.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:57.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:57.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:57.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836550 2026-02-22T21:08:57.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836550 2026-02-22T21:08:57.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550' 2026-02-22T21:08:57.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:57.175 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:57.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705732 2026-02-22T21:08:57.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705732 2026-02-22T21:08:57.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550 1-38654705732' 2026-02-22T21:08:57.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:57.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:57.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542209 2026-02-22T21:08:57.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542209 2026-02-22T21:08:57.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836550 1-38654705732 2-60129542209' 2026-02-22T21:08:57.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:57.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836550 2026-02-22T21:08:57.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:57.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:57.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836550 2026-02-22T21:08:57.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:57.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836550 2026-02-22T21:08:57.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836550' 2026-02-22T21:08:57.323 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836550 2026-02-22T21:08:57.323 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:57.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836548 -lt 21474836550 2026-02-22T21:08:57.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:08:58.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:08:58.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:58.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836551 -lt 21474836550 2026-02-22T21:08:58.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:58.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705732 2026-02-22T21:08:58.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:58.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:58.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705732 2026-02-22T21:08:58.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:58.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705732 2026-02-22T21:08:58.657 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705732 2026-02-22T21:08:58.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705732' 2026-02-22T21:08:58.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:58.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705732 -lt 38654705732 2026-02-22T21:08:58.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:58.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542209 2026-02-22T21:08:58.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:58.817 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:58.817 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542209 2026-02-22T21:08:58.818 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:58.819 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542209 2026-02-22T21:08:58.819 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542209 2026-02-22T21:08:58.820 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542209' 2026-02-22T21:08:58.820 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:08:58.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542210 -lt 60129542209 2026-02-22T21:08:58.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:08:58.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:08:58.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:08:59.127 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:08:59.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:08:59.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:08:59.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:08:59.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:08:59.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:08:59.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:08:59.291 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:08:59.291 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:08:59.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:08:59.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:59.292 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:08:59.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836554 2026-02-22T21:08:59.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836554 2026-02-22T21:08:59.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836554' 2026-02-22T21:08:59.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:59.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:08:59.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705735 2026-02-22T21:08:59.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705735 2026-02-22T21:08:59.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836554 1-38654705735' 2026-02-22T21:08:59.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:08:59.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:08:59.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542213 2026-02-22T21:08:59.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542213 2026-02-22T21:08:59.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836554 1-38654705735 2-60129542213' 2026-02-22T21:08:59.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:59.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836554 2026-02-22T21:08:59.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:59.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:08:59.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836554 2026-02-22T21:08:59.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:59.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836554 2026-02-22T21:08:59.516 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836554 2026-02-22T21:08:59.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836554' 2026-02-22T21:08:59.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:08:59.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836554 -lt 21474836554 2026-02-22T21:08:59.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:59.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705735 2026-02-22T21:08:59.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:59.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:08:59.686 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705735 2026-02-22T21:08:59.686 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:59.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705735 2026-02-22T21:08:59.687 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705735 2026-02-22T21:08:59.688 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705735' 2026-02-22T21:08:59.688 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:08:59.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705735 -lt 38654705735 2026-02-22T21:08:59.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:08:59.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542213 2026-02-22T21:08:59.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:08:59.854 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:08:59.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542213 2026-02-22T21:08:59.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:08:59.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542213 2026-02-22T21:08:59.857 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542213 2026-02-22T21:08:59.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542213' 2026-02-22T21:08:59.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:00.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542212 -lt 60129542213 2026-02-22T21:09:00.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:01.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:01.013 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:01.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542212 -lt 60129542213 2026-02-22T21:09:01.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:02.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-22T21:09:02.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:02.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542215 -lt 60129542213 2026-02-22T21:09:02.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:02.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:02.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:02.487 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:02.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:02.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:02.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:02.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:02.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:02.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:02.655 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:02.655 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:02.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:02.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:02.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:02.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836558 2026-02-22T21:09:02.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836558 2026-02-22T21:09:02.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558' 2026-02-22T21:09:02.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:02.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:02.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705739 2026-02-22T21:09:02.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705739 2026-02-22T21:09:02.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558 1-38654705739' 2026-02-22T21:09:02.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:02.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:02.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542217 2026-02-22T21:09:02.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542217 2026-02-22T21:09:02.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836558 1-38654705739 2-60129542217' 2026-02-22T21:09:02.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:02.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836558 2026-02-22T21:09:02.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:02.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:02.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836558 2026-02-22T21:09:02.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:02.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836558 2026-02-22T21:09:02.878 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836558 2026-02-22T21:09:02.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836558' 2026-02-22T21:09:02.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:03.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836556 -lt 21474836558 2026-02-22T21:09:03.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:04.047 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:04.048 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:04.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836559 -lt 21474836558 2026-02-22T21:09:04.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:04.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705739 2026-02-22T21:09:04.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:04.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:04.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705739 2026-02-22T21:09:04.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:04.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705739 2026-02-22T21:09:04.213 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705739' 2026-02-22T21:09:04.214 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705739 2026-02-22T21:09:04.214 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:04.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705740 -lt 38654705739 2026-02-22T21:09:04.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:04.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542217 2026-02-22T21:09:04.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:04.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:04.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542217 2026-02-22T21:09:04.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:04.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542217 2026-02-22T21:09:04.385 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542217 2026-02-22T21:09:04.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542217' 2026-02-22T21:09:04.386 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:04.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542218 -lt 60129542217 2026-02-22T21:09:04.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:04.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:04.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:04.691 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:04.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:04.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:04.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:04.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:04.703 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:04.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:04.866 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:04.866 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:04.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:04.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:04.867 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:04.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836561 2026-02-22T21:09:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836561 2026-02-22T21:09:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561' 2026-02-22T21:09:04.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:04.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705743 2026-02-22T21:09:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705743 2026-02-22T21:09:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561 1-38654705743' 2026-02-22T21:09:05.010 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:05.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:05.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542220 2026-02-22T21:09:05.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542220 2026-02-22T21:09:05.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836561 1-38654705743 2-60129542220' 2026-02-22T21:09:05.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:05.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836561 2026-02-22T21:09:05.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:05.084 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:05.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836561 2026-02-22T21:09:05.085 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:05.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836561 2026-02-22T21:09:05.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836561' 2026-02-22T21:09:05.087 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836561 2026-02-22T21:09:05.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:05.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836559 -lt 21474836561 2026-02-22T21:09:05.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:06.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:06.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:06.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836562 -lt 21474836561 2026-02-22T21:09:06.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:06.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705743 2026-02-22T21:09:06.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:06.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:06.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705743 2026-02-22T21:09:06.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:06.419 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705743 2026-02-22T21:09:06.419 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705743 2026-02-22T21:09:06.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705743' 2026-02-22T21:09:06.420 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:06.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705743 -lt 38654705743 2026-02-22T21:09:06.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:06.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542220 2026-02-22T21:09:06.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:06.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:06.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542220 2026-02-22T21:09:06.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:06.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542220 2026-02-22T21:09:06.591 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542220 2026-02-22T21:09:06.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542220' 2026-02-22T21:09:06.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:06.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542221 -lt 60129542220 2026-02-22T21:09:06.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:06.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:06.760 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:06.907 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:06.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:06.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:06.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:06.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:06.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:07.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:07.080 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:07.080 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:07.080 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:07.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:07.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:07.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836564 2026-02-22T21:09:07.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836564 2026-02-22T21:09:07.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564' 2026-02-22T21:09:07.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:07.151 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:07.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705746 2026-02-22T21:09:07.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705746 2026-02-22T21:09:07.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564 1-38654705746' 2026-02-22T21:09:07.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:07.223 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542223 2026-02-22T21:09:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542223 2026-02-22T21:09:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836564 1-38654705746 2-60129542223' 2026-02-22T21:09:07.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:07.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836564 2026-02-22T21:09:07.296 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:07.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:07.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836564 2026-02-22T21:09:07.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:07.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836564 2026-02-22T21:09:07.299 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836564 2026-02-22T21:09:07.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836564' 2026-02-22T21:09:07.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:07.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836562 -lt 21474836564 2026-02-22T21:09:07.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:08.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:08.466 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:08.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836565 -lt 21474836564 2026-02-22T21:09:08.627 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:08.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705746 2026-02-22T21:09:08.628 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:08.629 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:08.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705746 2026-02-22T21:09:08.630 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:08.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705746 2026-02-22T21:09:08.631 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705746 2026-02-22T21:09:08.632 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705746' 2026-02-22T21:09:08.632 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:08.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705746 -lt 38654705746 2026-02-22T21:09:08.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:08.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542223 2026-02-22T21:09:08.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:08.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:08.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542223 2026-02-22T21:09:08.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:08.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542223 2026-02-22T21:09:08.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542223' 2026-02-22T21:09:08.809 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542223 2026-02-22T21:09:08.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:08.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542224 -lt 60129542223 2026-02-22T21:09:08.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:08.969 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:08.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:09.118 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:09.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:09.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:09.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:09.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:09.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:09.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:09.279 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:09.279 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:09.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:09.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:09.280 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:09.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836567 2026-02-22T21:09:09.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836567 2026-02-22T21:09:09.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836567' 2026-02-22T21:09:09.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:09.351 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:09.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705749 2026-02-22T21:09:09.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705749 2026-02-22T21:09:09.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836567 1-38654705749' 2026-02-22T21:09:09.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:09.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:09.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542227 2026-02-22T21:09:09.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542227 2026-02-22T21:09:09.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836567 1-38654705749 2-60129542227' 2026-02-22T21:09:09.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:09.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836567 2026-02-22T21:09:09.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:09.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:09.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836567 2026-02-22T21:09:09.499 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:09.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836567 2026-02-22T21:09:09.501 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836567 2026-02-22T21:09:09.501 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2026-02-22T21:09:09.501 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:09.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836568 -lt 21474836567 2026-02-22T21:09:09.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:09.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705749 2026-02-22T21:09:09.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:09.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:09.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705749 2026-02-22T21:09:09.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:09.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705749 2026-02-22T21:09:09.664 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705749 2026-02-22T21:09:09.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705749' 2026-02-22T21:09:09.664 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:09.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705749 -lt 38654705749 2026-02-22T21:09:09.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:09.830 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542227 2026-02-22T21:09:09.830 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:09.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:09.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542227 2026-02-22T21:09:09.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:09.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542227 2026-02-22T21:09:09.834 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542227 2026-02-22T21:09:09.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542227' 2026-02-22T21:09:09.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:10.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542227 -lt 60129542227 2026-02-22T21:09:10.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:10.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:10.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:10.151 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:10.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:10.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:10.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:10.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:10.161 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:10.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:10.313 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:10.313 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:10.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:10.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:10.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:10.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836570 2026-02-22T21:09:10.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836570 2026-02-22T21:09:10.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836570' 2026-02-22T21:09:10.386 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:10.387 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:10.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705751 2026-02-22T21:09:10.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705751 2026-02-22T21:09:10.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836570 1-38654705751' 2026-02-22T21:09:10.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:10.459 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542229 2026-02-22T21:09:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542229 2026-02-22T21:09:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836570 1-38654705751 2-60129542229' 2026-02-22T21:09:10.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:10.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836570 2026-02-22T21:09:10.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:10.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:10.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836570 2026-02-22T21:09:10.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:10.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836570 2026-02-22T21:09:10.536 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836570 2026-02-22T21:09:10.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2026-02-22T21:09:10.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:10.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836568 -lt 21474836570 2026-02-22T21:09:10.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:11.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:11.704 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:11.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836571 -lt 21474836570 2026-02-22T21:09:11.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:11.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705751 2026-02-22T21:09:11.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:11.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:11.865 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705751 2026-02-22T21:09:11.865 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:11.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705751 2026-02-22T21:09:11.867 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705751 2026-02-22T21:09:11.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705751' 2026-02-22T21:09:11.867 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:12.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705752 -lt 38654705751 2026-02-22T21:09:12.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:12.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542229 2026-02-22T21:09:12.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:12.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:12.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542229 2026-02-22T21:09:12.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:12.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542229 2026-02-22T21:09:12.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542229' 2026-02-22T21:09:12.041 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542229 2026-02-22T21:09:12.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:12.202 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542230 -lt 60129542229 2026-02-22T21:09:12.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:12.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:12.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:12.349 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:12.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:12.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:12.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:12.361 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:12.361 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:12.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:12.516 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:12.516 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:12.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:12.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:12.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:12.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836573 2026-02-22T21:09:12.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836573 2026-02-22T21:09:12.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836573' 2026-02-22T21:09:12.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:12.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:12.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705754 2026-02-22T21:09:12.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705754 2026-02-22T21:09:12.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836573 1-38654705754' 2026-02-22T21:09:12.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:12.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:12.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542232 2026-02-22T21:09:12.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542232 2026-02-22T21:09:12.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836573 1-38654705754 2-60129542232' 2026-02-22T21:09:12.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:12.732 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836573 2026-02-22T21:09:12.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:12.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:12.734 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836573 2026-02-22T21:09:12.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:12.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836573 2026-02-22T21:09:12.736 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836573 2026-02-22T21:09:12.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2026-02-22T21:09:12.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:12.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836571 -lt 21474836573 2026-02-22T21:09:12.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:13.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:13.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:14.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836574 -lt 21474836573 2026-02-22T21:09:14.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:14.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705754 2026-02-22T21:09:14.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:14.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:14.052 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705754 2026-02-22T21:09:14.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:14.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705754 2026-02-22T21:09:14.054 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705754 2026-02-22T21:09:14.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705754' 2026-02-22T21:09:14.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:14.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705755 -lt 38654705754 2026-02-22T21:09:14.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:14.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542232 2026-02-22T21:09:14.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:14.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:14.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542232 2026-02-22T21:09:14.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:14.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542232 2026-02-22T21:09:14.226 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542232 2026-02-22T21:09:14.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542232' 2026-02-22T21:09:14.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:14.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542233 -lt 60129542232 2026-02-22T21:09:14.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:14.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:14.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:14.540 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:14.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:14.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:14.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:14.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:14.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:14.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:14.704 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:14.704 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:14.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:14.704 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:14.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:14.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836576 2026-02-22T21:09:14.774 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836576 2026-02-22T21:09:14.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836576' 2026-02-22T21:09:14.775 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:14.775 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:14.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705757 2026-02-22T21:09:14.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705757 2026-02-22T21:09:14.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836576 1-38654705757' 2026-02-22T21:09:14.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:14.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:14.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542235 2026-02-22T21:09:14.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542235 2026-02-22T21:09:14.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836576 1-38654705757 2-60129542235' 2026-02-22T21:09:14.918 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:14.918 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836576 2026-02-22T21:09:14.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:14.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:14.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836576 2026-02-22T21:09:14.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:14.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836576 2026-02-22T21:09:14.923 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836576 2026-02-22T21:09:14.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836576' 2026-02-22T21:09:14.923 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:15.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836574 -lt 21474836576 2026-02-22T21:09:15.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:16.082 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:16.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:16.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836577 -lt 21474836576 2026-02-22T21:09:16.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:16.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705757 2026-02-22T21:09:16.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:16.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:16.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705757 2026-02-22T21:09:16.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:16.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705757 2026-02-22T21:09:16.248 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705757 2026-02-22T21:09:16.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705757' 2026-02-22T21:09:16.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:16.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705758 -lt 38654705757 2026-02-22T21:09:16.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:16.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542235 2026-02-22T21:09:16.408 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:16.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:16.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542235 2026-02-22T21:09:16.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:16.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542235 2026-02-22T21:09:16.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542235' 2026-02-22T21:09:16.411 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542235 2026-02-22T21:09:16.412 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:16.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542236 -lt 60129542235 2026-02-22T21:09:16.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:16.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:16.581 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:16.727 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:16.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:16.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:16.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:16.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:16.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:16.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:16.898 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:16.898 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:16.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:16.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:16.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:16.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836579 2026-02-22T21:09:16.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836579 2026-02-22T21:09:16.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836579' 2026-02-22T21:09:16.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:16.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:17.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705761 2026-02-22T21:09:17.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705761 2026-02-22T21:09:17.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836579 1-38654705761' 2026-02-22T21:09:17.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:17.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:17.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542238 2026-02-22T21:09:17.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542238 2026-02-22T21:09:17.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836579 1-38654705761 2-60129542238' 2026-02-22T21:09:17.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:17.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836579 2026-02-22T21:09:17.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:17.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:17.121 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836579 2026-02-22T21:09:17.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:17.122 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836579 2026-02-22T21:09:17.123 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836579 2026-02-22T21:09:17.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836579' 2026-02-22T21:09:17.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:17.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836577 -lt 21474836579 2026-02-22T21:09:17.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:18.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:18.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:18.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836580 -lt 21474836579 2026-02-22T21:09:18.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:18.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705761 2026-02-22T21:09:18.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:18.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:18.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705761 2026-02-22T21:09:18.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:18.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705761 2026-02-22T21:09:18.449 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705761 2026-02-22T21:09:18.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705761' 2026-02-22T21:09:18.449 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:18.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705761 -lt 38654705761 2026-02-22T21:09:18.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:18.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542238 2026-02-22T21:09:18.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:18.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:18.610 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542238 2026-02-22T21:09:18.610 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:18.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542238 2026-02-22T21:09:18.611 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542238 2026-02-22T21:09:18.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542238' 2026-02-22T21:09:18.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:18.781 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542239 -lt 60129542238 2026-02-22T21:09:18.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:18.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:18.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:18.928 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:18.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:18.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:18.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:18.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:18.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:19.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:19.091 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:19.091 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:19.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:19.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:19.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:19.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836582 2026-02-22T21:09:19.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836582 2026-02-22T21:09:19.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582' 2026-02-22T21:09:19.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:19.164 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:19.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705764 2026-02-22T21:09:19.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705764 2026-02-22T21:09:19.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582 1-38654705764' 2026-02-22T21:09:19.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:19.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:19.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542241 2026-02-22T21:09:19.307 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542241 2026-02-22T21:09:19.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836582 1-38654705764 2-60129542241' 2026-02-22T21:09:19.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:19.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836582 2026-02-22T21:09:19.308 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:19.309 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:19.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836582 2026-02-22T21:09:19.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:19.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836582 2026-02-22T21:09:19.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836582' 2026-02-22T21:09:19.312 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836582 2026-02-22T21:09:19.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:19.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836580 -lt 21474836582 2026-02-22T21:09:19.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:20.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:20.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:20.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836583 -lt 21474836582 2026-02-22T21:09:20.631 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:20.631 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705764 2026-02-22T21:09:20.631 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:20.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:20.633 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705764 2026-02-22T21:09:20.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:20.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705764 2026-02-22T21:09:20.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705764' 2026-02-22T21:09:20.635 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705764 2026-02-22T21:09:20.635 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:20.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705764 -lt 38654705764 2026-02-22T21:09:20.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:20.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542241 2026-02-22T21:09:20.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:20.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:20.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542241 2026-02-22T21:09:20.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:20.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542241 2026-02-22T21:09:20.807 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542241 2026-02-22T21:09:20.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542241' 2026-02-22T21:09:20.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:20.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542242 -lt 60129542241 2026-02-22T21:09:20.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:20.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:20.974 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:21.126 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:21.135 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:21.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:21.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:21.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:21.136 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:21.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:21.296 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:21.296 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:21.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:21.296 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:21.297 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:21.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836586 2026-02-22T21:09:21.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836586 2026-02-22T21:09:21.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836586' 2026-02-22T21:09:21.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:21.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:21.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705767 2026-02-22T21:09:21.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705767 2026-02-22T21:09:21.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836586 1-38654705767' 2026-02-22T21:09:21.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:21.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:21.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542245 2026-02-22T21:09:21.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542245 2026-02-22T21:09:21.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836586 1-38654705767 2-60129542245' 2026-02-22T21:09:21.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:21.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836586 2026-02-22T21:09:21.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:21.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:21.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836586 2026-02-22T21:09:21.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:21.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836586 2026-02-22T21:09:21.518 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836586 2026-02-22T21:09:21.519 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836586' 2026-02-22T21:09:21.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:21.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836586 -lt 21474836586 2026-02-22T21:09:21.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:21.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705767 2026-02-22T21:09:21.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:21.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:21.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705767 2026-02-22T21:09:21.686 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:21.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705767 2026-02-22T21:09:21.687 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705767 2026-02-22T21:09:21.687 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705767' 2026-02-22T21:09:21.687 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:21.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705767 -lt 38654705767 2026-02-22T21:09:21.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:21.848 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542245 2026-02-22T21:09:21.849 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:21.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:21.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542245 2026-02-22T21:09:21.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:21.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542245 2026-02-22T21:09:21.852 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542245 2026-02-22T21:09:21.852 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542245' 2026-02-22T21:09:21.852 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:22.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542244 -lt 60129542245 2026-02-22T21:09:22.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:23.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:23.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:23.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542244 -lt 60129542245 2026-02-22T21:09:23.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:24.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-22T21:09:24.186 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:24.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542247 -lt 60129542245 2026-02-22T21:09:24.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:24.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:24.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:24.492 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:24.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:24.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:24.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:24.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:24.503 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:24.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:24.662 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:24.662 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:24.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:24.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:24.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:24.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836590 2026-02-22T21:09:24.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836590 2026-02-22T21:09:24.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836590' 2026-02-22T21:09:24.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:24.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:24.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705771 2026-02-22T21:09:24.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705771 2026-02-22T21:09:24.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836590 1-38654705771' 2026-02-22T21:09:24.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:24.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:24.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542249 2026-02-22T21:09:24.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542249 2026-02-22T21:09:24.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836590 1-38654705771 2-60129542249' 2026-02-22T21:09:24.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:24.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836590 2026-02-22T21:09:24.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:24.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:24.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836590 2026-02-22T21:09:24.881 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:24.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836590 2026-02-22T21:09:24.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836590' 2026-02-22T21:09:24.883 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836590 2026-02-22T21:09:24.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:25.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836588 -lt 21474836590 2026-02-22T21:09:25.048 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:26.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:26.050 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:26.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836591 -lt 21474836590 2026-02-22T21:09:26.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:26.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705771 2026-02-22T21:09:26.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:26.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:26.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705771 2026-02-22T21:09:26.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:26.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705771 2026-02-22T21:09:26.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705771' 2026-02-22T21:09:26.210 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705771 2026-02-22T21:09:26.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:26.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705772 -lt 38654705771 2026-02-22T21:09:26.376 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:26.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542249 2026-02-22T21:09:26.377 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:26.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:26.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542249 2026-02-22T21:09:26.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:26.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542249 2026-02-22T21:09:26.381 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542249 2026-02-22T21:09:26.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542249' 2026-02-22T21:09:26.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:26.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542250 -lt 60129542249 2026-02-22T21:09:26.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:26.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:26.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:26.688 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:26.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:26.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:26.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:26.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:26.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:26.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:26.855 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:26.855 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:26.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:26.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:26.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:26.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836593 2026-02-22T21:09:26.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836593 2026-02-22T21:09:26.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836593' 2026-02-22T21:09:26.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:26.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:26.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705775 2026-02-22T21:09:26.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705775 2026-02-22T21:09:26.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836593 1-38654705775' 2026-02-22T21:09:26.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:26.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:27.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542252 2026-02-22T21:09:27.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542252 2026-02-22T21:09:27.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836593 1-38654705775 2-60129542252' 2026-02-22T21:09:27.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:27.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836593 2026-02-22T21:09:27.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:27.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:27.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836593 2026-02-22T21:09:27.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:27.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836593 2026-02-22T21:09:27.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2026-02-22T21:09:27.076 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836593 2026-02-22T21:09:27.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:27.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836591 -lt 21474836593 2026-02-22T21:09:27.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:28.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:28.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:28.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836594 -lt 21474836593 2026-02-22T21:09:28.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:28.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705775 2026-02-22T21:09:28.405 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:28.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:28.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705775 2026-02-22T21:09:28.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:28.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705775 2026-02-22T21:09:28.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705775' 2026-02-22T21:09:28.408 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705775 2026-02-22T21:09:28.409 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:28.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705775 -lt 38654705775 2026-02-22T21:09:28.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:28.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542252 2026-02-22T21:09:28.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:28.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:28.578 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542252 2026-02-22T21:09:28.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:28.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542252 2026-02-22T21:09:28.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542252' 2026-02-22T21:09:28.581 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542252 2026-02-22T21:09:28.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:28.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542253 -lt 60129542252 2026-02-22T21:09:28.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:28.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:28.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:28.894 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:28.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:28.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:28.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:28.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:28.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:29.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:29.058 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:29.058 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:29.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:29.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:29.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:29.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836596 2026-02-22T21:09:29.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836596 2026-02-22T21:09:29.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596' 2026-02-22T21:09:29.130 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:29.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:29.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705778 2026-02-22T21:09:29.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705778 2026-02-22T21:09:29.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596 1-38654705778' 2026-02-22T21:09:29.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:29.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:29.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542255 2026-02-22T21:09:29.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542255 2026-02-22T21:09:29.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836596 1-38654705778 2-60129542255' 2026-02-22T21:09:29.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:29.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836596 2026-02-22T21:09:29.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:29.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:29.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836596 2026-02-22T21:09:29.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:29.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836596 2026-02-22T21:09:29.280 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836596 2026-02-22T21:09:29.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836596' 2026-02-22T21:09:29.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:29.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836594 -lt 21474836596 2026-02-22T21:09:29.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:30.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:30.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:30.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836597 -lt 21474836596 2026-02-22T21:09:30.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:30.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705778 2026-02-22T21:09:30.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:30.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:30.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705778 2026-02-22T21:09:30.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:30.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705778 2026-02-22T21:09:30.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705778' 2026-02-22T21:09:30.608 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705778 2026-02-22T21:09:30.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:30.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705778 -lt 38654705778 2026-02-22T21:09:30.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:30.779 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542255 2026-02-22T21:09:30.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:30.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:30.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542255 2026-02-22T21:09:30.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:30.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542255 2026-02-22T21:09:30.783 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542255 2026-02-22T21:09:30.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542255' 2026-02-22T21:09:30.784 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:30.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542256 -lt 60129542255 2026-02-22T21:09:30.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:30.952 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:30.953 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:31.097 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:31.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:31.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:31.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:31.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:31.106 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:31.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:31.268 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:31.268 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:31.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:31.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:31.269 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:31.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836599 2026-02-22T21:09:31.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836599 2026-02-22T21:09:31.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836599' 2026-02-22T21:09:31.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:31.343 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:31.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705781 2026-02-22T21:09:31.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705781 2026-02-22T21:09:31.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836599 1-38654705781' 2026-02-22T21:09:31.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:31.416 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:31.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542259 2026-02-22T21:09:31.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542259 2026-02-22T21:09:31.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836599 1-38654705781 2-60129542259' 2026-02-22T21:09:31.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:31.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836599 2026-02-22T21:09:31.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:31.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:31.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836599 2026-02-22T21:09:31.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:31.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836599 2026-02-22T21:09:31.491 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836599 2026-02-22T21:09:31.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836599' 2026-02-22T21:09:31.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:31.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836600 -lt 21474836599 2026-02-22T21:09:31.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:31.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705781 2026-02-22T21:09:31.654 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:31.656 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:31.656 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705781 2026-02-22T21:09:31.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:31.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705781 2026-02-22T21:09:31.658 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705781 2026-02-22T21:09:31.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705781' 2026-02-22T21:09:31.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:31.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705781 -lt 38654705781 2026-02-22T21:09:31.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:31.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542259 2026-02-22T21:09:31.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:31.827 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:31.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542259 2026-02-22T21:09:31.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:31.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542259 2026-02-22T21:09:31.829 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542259 2026-02-22T21:09:31.829 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542259' 2026-02-22T21:09:31.829 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:31.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542259 -lt 60129542259 2026-02-22T21:09:31.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:31.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:31.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:32.134 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:32.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:32.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:32.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:32.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:32.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:32.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:32.298 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:32.299 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:32.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:32.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:32.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:32.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836602 2026-02-22T21:09:32.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836602 2026-02-22T21:09:32.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836602' 2026-02-22T21:09:32.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:32.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:32.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705783 2026-02-22T21:09:32.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705783 2026-02-22T21:09:32.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836602 1-38654705783' 2026-02-22T21:09:32.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:32.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:32.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542261 2026-02-22T21:09:32.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542261 2026-02-22T21:09:32.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836602 1-38654705783 2-60129542261' 2026-02-22T21:09:32.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:32.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836602 2026-02-22T21:09:32.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:32.520 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:32.520 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836602 2026-02-22T21:09:32.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:32.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836602 2026-02-22T21:09:32.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836602' 2026-02-22T21:09:32.522 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836602 2026-02-22T21:09:32.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:32.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836600 -lt 21474836602 2026-02-22T21:09:32.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:33.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:33.690 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:33.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836603 -lt 21474836602 2026-02-22T21:09:33.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:33.853 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705783 2026-02-22T21:09:33.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:33.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:33.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705783 2026-02-22T21:09:33.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:33.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705783 2026-02-22T21:09:33.857 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705783 2026-02-22T21:09:33.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705783' 2026-02-22T21:09:33.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:34.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705784 -lt 38654705783 2026-02-22T21:09:34.021 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:34.021 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542261 2026-02-22T21:09:34.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:34.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:34.023 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542261 2026-02-22T21:09:34.024 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:34.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542261 2026-02-22T21:09:34.025 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542261 2026-02-22T21:09:34.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542261' 2026-02-22T21:09:34.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:34.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542262 -lt 60129542261 2026-02-22T21:09:34.185 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:34.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:34.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:34.334 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:34.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:34.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:34.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:34.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:34.344 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:34.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:34.502 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:34.502 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:34.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:34.502 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:34.502 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:34.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836605 2026-02-22T21:09:34.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836605 2026-02-22T21:09:34.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836605' 2026-02-22T21:09:34.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:34.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:34.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705786 2026-02-22T21:09:34.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705786 2026-02-22T21:09:34.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836605 1-38654705786' 2026-02-22T21:09:34.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:34.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:34.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542264 2026-02-22T21:09:34.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542264 2026-02-22T21:09:34.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836605 1-38654705786 2-60129542264' 2026-02-22T21:09:34.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:34.714 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836605 2026-02-22T21:09:34.714 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:34.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:34.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836605 2026-02-22T21:09:34.716 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:34.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836605 2026-02-22T21:09:34.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836605' 2026-02-22T21:09:34.718 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836605 2026-02-22T21:09:34.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:34.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836603 -lt 21474836605 2026-02-22T21:09:34.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:35.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:35.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:36.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836606 -lt 21474836605 2026-02-22T21:09:36.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:36.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705786 2026-02-22T21:09:36.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:36.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:36.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705786 2026-02-22T21:09:36.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:36.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705786 2026-02-22T21:09:36.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705786' 2026-02-22T21:09:36.038 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705786 2026-02-22T21:09:36.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:36.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705787 -lt 38654705786 2026-02-22T21:09:36.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:36.205 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542264 2026-02-22T21:09:36.206 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:36.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:36.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542264 2026-02-22T21:09:36.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:36.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542264 2026-02-22T21:09:36.209 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542264 2026-02-22T21:09:36.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542264' 2026-02-22T21:09:36.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:36.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542265 -lt 60129542264 2026-02-22T21:09:36.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:36.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:36.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:36.512 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:36.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:36.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:36.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:36.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:36.522 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:36.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:36.679 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:36.679 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:36.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:36.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:36.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:36.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836608 2026-02-22T21:09:36.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836608 2026-02-22T21:09:36.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836608' 2026-02-22T21:09:36.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:36.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:36.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705789 2026-02-22T21:09:36.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705789 2026-02-22T21:09:36.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836608 1-38654705789' 2026-02-22T21:09:36.823 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:36.823 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:36.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542267 2026-02-22T21:09:36.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542267 2026-02-22T21:09:36.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836608 1-38654705789 2-60129542267' 2026-02-22T21:09:36.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:36.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836608 2026-02-22T21:09:36.898 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:36.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:36.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836608 2026-02-22T21:09:36.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:36.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836608 2026-02-22T21:09:36.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836608' 2026-02-22T21:09:36.901 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836608 2026-02-22T21:09:36.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:37.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836606 -lt 21474836608 2026-02-22T21:09:37.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:38.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:38.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:38.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836609 -lt 21474836608 2026-02-22T21:09:38.215 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:38.215 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705789 2026-02-22T21:09:38.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:38.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:38.218 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705789 2026-02-22T21:09:38.218 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:38.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705789 2026-02-22T21:09:38.219 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705789 2026-02-22T21:09:38.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705789' 2026-02-22T21:09:38.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:38.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705790 -lt 38654705789 2026-02-22T21:09:38.387 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:38.388 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542267 2026-02-22T21:09:38.388 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:38.389 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:38.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542267 2026-02-22T21:09:38.390 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:38.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542267 2026-02-22T21:09:38.391 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542267 2026-02-22T21:09:38.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542267' 2026-02-22T21:09:38.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:38.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542268 -lt 60129542267 2026-02-22T21:09:38.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:38.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:38.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:38.706 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:38.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:38.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:38.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:38.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:38.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:38.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:38.870 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:38.870 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:38.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:38.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:38.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:38.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836611 2026-02-22T21:09:38.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836611 2026-02-22T21:09:38.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836611' 2026-02-22T21:09:38.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:38.945 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:39.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705793 2026-02-22T21:09:39.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705793 2026-02-22T21:09:39.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836611 1-38654705793' 2026-02-22T21:09:39.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:39.014 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:39.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542270 2026-02-22T21:09:39.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542270 2026-02-22T21:09:39.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836611 1-38654705793 2-60129542270' 2026-02-22T21:09:39.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:39.088 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836611 2026-02-22T21:09:39.088 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:39.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:39.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836611 2026-02-22T21:09:39.090 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:39.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836611 2026-02-22T21:09:39.091 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836611 2026-02-22T21:09:39.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836611' 2026-02-22T21:09:39.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:39.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836609 -lt 21474836611 2026-02-22T21:09:39.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:40.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:40.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:40.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836612 -lt 21474836611 2026-02-22T21:09:40.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:40.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705793 2026-02-22T21:09:40.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:40.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:40.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705793 2026-02-22T21:09:40.419 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:40.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705793 2026-02-22T21:09:40.420 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705793 2026-02-22T21:09:40.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705793' 2026-02-22T21:09:40.421 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:40.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705793 -lt 38654705793 2026-02-22T21:09:40.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:40.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542270 2026-02-22T21:09:40.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:40.580 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:40.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542270 2026-02-22T21:09:40.581 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:40.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542270 2026-02-22T21:09:40.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542270' 2026-02-22T21:09:40.583 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542270 2026-02-22T21:09:40.583 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:40.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542271 -lt 60129542270 2026-02-22T21:09:40.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:40.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:40.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:40.898 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:40.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:40.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:40.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:40.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:40.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:41.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:41.070 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:41.070 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:41.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:41.070 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:41.070 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:41.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836614 2026-02-22T21:09:41.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836614 2026-02-22T21:09:41.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836614' 2026-02-22T21:09:41.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:41.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:41.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705796 2026-02-22T21:09:41.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705796 2026-02-22T21:09:41.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836614 1-38654705796' 2026-02-22T21:09:41.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:41.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:41.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542273 2026-02-22T21:09:41.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542273 2026-02-22T21:09:41.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836614 1-38654705796 2-60129542273' 2026-02-22T21:09:41.289 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:41.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836614 2026-02-22T21:09:41.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:41.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:41.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836614 2026-02-22T21:09:41.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:41.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836614 2026-02-22T21:09:41.293 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836614 2026-02-22T21:09:41.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836614' 2026-02-22T21:09:41.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:41.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836612 -lt 21474836614 2026-02-22T21:09:41.452 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:42.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:42.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:42.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836615 -lt 21474836614 2026-02-22T21:09:42.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:42.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705796 2026-02-22T21:09:42.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:42.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:42.613 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705796 2026-02-22T21:09:42.613 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:42.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705796 2026-02-22T21:09:42.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705796' 2026-02-22T21:09:42.615 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705796 2026-02-22T21:09:42.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:42.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705796 -lt 38654705796 2026-02-22T21:09:42.777 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:42.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542273 2026-02-22T21:09:42.778 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:42.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:42.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542273 2026-02-22T21:09:42.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:42.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542273 2026-02-22T21:09:42.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542273' 2026-02-22T21:09:42.782 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542273 2026-02-22T21:09:42.782 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:42.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542274 -lt 60129542273 2026-02-22T21:09:42.949 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:42.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:42.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:43.098 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:43.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:43.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:43.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:43.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:43.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:43.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:43.266 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:43.266 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:43.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:43.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:43.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:43.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836617 2026-02-22T21:09:43.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836617 2026-02-22T21:09:43.341 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617' 2026-02-22T21:09:43.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:43.342 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:43.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705799 2026-02-22T21:09:43.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705799 2026-02-22T21:09:43.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617 1-38654705799' 2026-02-22T21:09:43.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:43.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:43.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542277 2026-02-22T21:09:43.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542277 2026-02-22T21:09:43.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836617 1-38654705799 2-60129542277' 2026-02-22T21:09:43.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:43.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836617 2026-02-22T21:09:43.484 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:43.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:43.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836617 2026-02-22T21:09:43.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:43.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836617 2026-02-22T21:09:43.487 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836617' 2026-02-22T21:09:43.488 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836617 2026-02-22T21:09:43.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:43.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836618 -lt 21474836617 2026-02-22T21:09:43.658 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:43.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705799 2026-02-22T21:09:43.659 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:43.660 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:43.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705799 2026-02-22T21:09:43.661 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:43.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705799 2026-02-22T21:09:43.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705799' 2026-02-22T21:09:43.662 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705799 2026-02-22T21:09:43.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:43.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705799 -lt 38654705799 2026-02-22T21:09:43.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:43.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542277 2026-02-22T21:09:43.832 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:43.833 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:43.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542277 2026-02-22T21:09:43.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:43.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542277 2026-02-22T21:09:43.836 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542277 2026-02-22T21:09:43.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542277' 2026-02-22T21:09:43.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:44.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542277 -lt 60129542277 2026-02-22T21:09:44.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:44.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:44.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:44.152 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:44.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:44.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:44.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:44.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:44.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:44.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:44.320 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:44.320 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:44.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:44.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:44.321 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:44.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836620 2026-02-22T21:09:44.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836620 2026-02-22T21:09:44.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836620' 2026-02-22T21:09:44.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:44.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:44.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705801 2026-02-22T21:09:44.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705801 2026-02-22T21:09:44.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836620 1-38654705801' 2026-02-22T21:09:44.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:44.464 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:44.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542279 2026-02-22T21:09:44.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542279 2026-02-22T21:09:44.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836620 1-38654705801 2-60129542279' 2026-02-22T21:09:44.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:44.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836620 2026-02-22T21:09:44.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:44.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:44.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836620 2026-02-22T21:09:44.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:44.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836620 2026-02-22T21:09:44.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836620' 2026-02-22T21:09:44.538 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836620 2026-02-22T21:09:44.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:44.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836618 -lt 21474836620 2026-02-22T21:09:44.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:45.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:45.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:45.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836621 -lt 21474836620 2026-02-22T21:09:45.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:45.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705801 2026-02-22T21:09:45.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:45.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:45.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705801 2026-02-22T21:09:45.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:45.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705801 2026-02-22T21:09:45.860 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705801 2026-02-22T21:09:45.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705801' 2026-02-22T21:09:45.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:46.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705802 -lt 38654705801 2026-02-22T21:09:46.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:46.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542279 2026-02-22T21:09:46.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:46.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:46.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542279 2026-02-22T21:09:46.032 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:46.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542279 2026-02-22T21:09:46.034 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542279 2026-02-22T21:09:46.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542279' 2026-02-22T21:09:46.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:46.197 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542280 -lt 60129542279 2026-02-22T21:09:46.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:46.198 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:46.199 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:46.347 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:46.356 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:46.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:46.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:46.357 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:46.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:46.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:46.516 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:46.516 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:46.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:46.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:46.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:46.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836623 2026-02-22T21:09:46.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836623 2026-02-22T21:09:46.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836623' 2026-02-22T21:09:46.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:46.587 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:46.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705804 2026-02-22T21:09:46.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705804 2026-02-22T21:09:46.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836623 1-38654705804' 2026-02-22T21:09:46.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:46.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:46.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542282 2026-02-22T21:09:46.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542282 2026-02-22T21:09:46.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836623 1-38654705804 2-60129542282' 2026-02-22T21:09:46.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:46.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836623 2026-02-22T21:09:46.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:46.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:46.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836623 2026-02-22T21:09:46.733 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:46.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836623 2026-02-22T21:09:46.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836623' 2026-02-22T21:09:46.735 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836623 2026-02-22T21:09:46.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:46.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836621 -lt 21474836623 2026-02-22T21:09:46.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:47.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:47.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:48.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836624 -lt 21474836623 2026-02-22T21:09:48.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:48.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705804 2026-02-22T21:09:48.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:48.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:48.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705804 2026-02-22T21:09:48.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:48.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705804 2026-02-22T21:09:48.063 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705804' 2026-02-22T21:09:48.064 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705804 2026-02-22T21:09:48.064 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:48.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705805 -lt 38654705804 2026-02-22T21:09:48.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:48.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542282 2026-02-22T21:09:48.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:48.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:48.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542282 2026-02-22T21:09:48.238 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:48.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542282 2026-02-22T21:09:48.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542282' 2026-02-22T21:09:48.240 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542282 2026-02-22T21:09:48.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:48.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542283 -lt 60129542282 2026-02-22T21:09:48.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:48.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:48.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:48.551 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:48.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:48.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:48.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:48.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:48.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:48.714 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:48.715 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:48.715 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:48.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:48.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:48.715 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:48.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836626 2026-02-22T21:09:48.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836626 2026-02-22T21:09:48.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836626' 2026-02-22T21:09:48.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:48.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:48.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705807 2026-02-22T21:09:48.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705807 2026-02-22T21:09:48.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836626 1-38654705807' 2026-02-22T21:09:48.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:48.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:48.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542285 2026-02-22T21:09:48.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542285 2026-02-22T21:09:48.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836626 1-38654705807 2-60129542285' 2026-02-22T21:09:48.931 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:48.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836626 2026-02-22T21:09:48.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:48.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:48.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836626 2026-02-22T21:09:48.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:48.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836626 2026-02-22T21:09:48.935 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836626 2026-02-22T21:09:48.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836626' 2026-02-22T21:09:48.935 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:49.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836624 -lt 21474836626 2026-02-22T21:09:49.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:50.100 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:50.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:50.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836627 -lt 21474836626 2026-02-22T21:09:50.260 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:50.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705807 2026-02-22T21:09:50.261 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:50.261 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:50.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705807 2026-02-22T21:09:50.262 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:50.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705807 2026-02-22T21:09:50.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705807' 2026-02-22T21:09:50.264 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705807 2026-02-22T21:09:50.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:50.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705808 -lt 38654705807 2026-02-22T21:09:50.423 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:50.423 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542285 2026-02-22T21:09:50.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:50.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:50.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542285 2026-02-22T21:09:50.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:50.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542285 2026-02-22T21:09:50.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542285' 2026-02-22T21:09:50.428 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542285 2026-02-22T21:09:50.428 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:50.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542286 -lt 60129542285 2026-02-22T21:09:50.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:50.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:50.589 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:50.732 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:50.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:50.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:50.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:50.741 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:50.741 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:50.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:50.900 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:50.900 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:50.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:50.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:50.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:50.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836629 2026-02-22T21:09:50.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836629 2026-02-22T21:09:50.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836629' 2026-02-22T21:09:50.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:50.972 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:51.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705811 2026-02-22T21:09:51.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705811 2026-02-22T21:09:51.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836629 1-38654705811' 2026-02-22T21:09:51.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:51.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:51.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542288 2026-02-22T21:09:51.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542288 2026-02-22T21:09:51.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836629 1-38654705811 2-60129542288' 2026-02-22T21:09:51.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:51.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836629 2026-02-22T21:09:51.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:51.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:51.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836629 2026-02-22T21:09:51.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:51.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836629 2026-02-22T21:09:51.119 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836629 2026-02-22T21:09:51.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836629' 2026-02-22T21:09:51.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:51.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836627 -lt 21474836629 2026-02-22T21:09:51.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:52.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:52.285 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:52.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836630 -lt 21474836629 2026-02-22T21:09:52.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:52.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705811 2026-02-22T21:09:52.447 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:52.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:52.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705811 2026-02-22T21:09:52.448 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:52.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705811 2026-02-22T21:09:52.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705811' 2026-02-22T21:09:52.450 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705811 2026-02-22T21:09:52.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:52.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705811 -lt 38654705811 2026-02-22T21:09:52.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:52.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542288 2026-02-22T21:09:52.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:52.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:52.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542288 2026-02-22T21:09:52.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:52.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542288 2026-02-22T21:09:52.619 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542288 2026-02-22T21:09:52.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542288' 2026-02-22T21:09:52.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:52.786 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542289 -lt 60129542288 2026-02-22T21:09:52.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:52.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:52.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:52.937 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:52.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:52.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:52.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:52.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:52.948 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:53.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:53.106 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:53.106 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:53.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:53.106 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:53.106 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:53.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836632 2026-02-22T21:09:53.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836632 2026-02-22T21:09:53.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836632' 2026-02-22T21:09:53.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:53.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:53.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705814 2026-02-22T21:09:53.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705814 2026-02-22T21:09:53.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836632 1-38654705814' 2026-02-22T21:09:53.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:53.252 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:53.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542291 2026-02-22T21:09:53.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542291 2026-02-22T21:09:53.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836632 1-38654705814 2-60129542291' 2026-02-22T21:09:53.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:53.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836632 2026-02-22T21:09:53.322 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:53.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:53.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836632 2026-02-22T21:09:53.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:53.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836632 2026-02-22T21:09:53.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836632' 2026-02-22T21:09:53.326 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836632 2026-02-22T21:09:53.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:53.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836630 -lt 21474836632 2026-02-22T21:09:53.498 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:54.499 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:54.500 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:54.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836633 -lt 21474836632 2026-02-22T21:09:54.657 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:54.657 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705814 2026-02-22T21:09:54.658 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:54.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:54.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705814 2026-02-22T21:09:54.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:54.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705814 2026-02-22T21:09:54.661 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705814' 2026-02-22T21:09:54.661 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705814 2026-02-22T21:09:54.662 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:54.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705814 -lt 38654705814 2026-02-22T21:09:54.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:54.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542291 2026-02-22T21:09:54.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:54.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:54.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542291 2026-02-22T21:09:54.837 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:54.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542291 2026-02-22T21:09:54.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542291' 2026-02-22T21:09:54.839 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542291 2026-02-22T21:09:54.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:55.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542292 -lt 60129542291 2026-02-22T21:09:55.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:55.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:55.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:55.151 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:55.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:55.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:55.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:55.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:55.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:55.319 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:55.319 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:55.319 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:55.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:55.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:55.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:55.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836636 2026-02-22T21:09:55.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836636 2026-02-22T21:09:55.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836636' 2026-02-22T21:09:55.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:55.393 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:55.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705817 2026-02-22T21:09:55.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705817 2026-02-22T21:09:55.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836636 1-38654705817' 2026-02-22T21:09:55.463 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:55.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:55.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542295 2026-02-22T21:09:55.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542295 2026-02-22T21:09:55.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836636 1-38654705817 2-60129542295' 2026-02-22T21:09:55.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:55.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836636 2026-02-22T21:09:55.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:55.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:55.537 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836636 2026-02-22T21:09:55.537 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:55.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836636 2026-02-22T21:09:55.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836636' 2026-02-22T21:09:55.539 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836636 2026-02-22T21:09:55.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:55.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836636 -lt 21474836636 2026-02-22T21:09:55.707 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:55.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705817 2026-02-22T21:09:55.708 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:55.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:09:55.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705817 2026-02-22T21:09:55.710 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:55.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705817 2026-02-22T21:09:55.712 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705817' 2026-02-22T21:09:55.712 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705817 2026-02-22T21:09:55.712 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:09:55.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705817 -lt 38654705817 2026-02-22T21:09:55.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:55.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542295 2026-02-22T21:09:55.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:55.884 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:09:55.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542295 2026-02-22T21:09:55.885 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:55.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542295 2026-02-22T21:09:55.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542295' 2026-02-22T21:09:55.887 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542295 2026-02-22T21:09:55.887 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:56.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542294 -lt 60129542295 2026-02-22T21:09:56.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:57.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:09:57.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:57.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542294 -lt 60129542295 2026-02-22T21:09:57.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:09:58.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-22T21:09:58.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:09:58.377 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542297 -lt 60129542295 2026-02-22T21:09:58.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:09:58.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:09:58.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:09:58.523 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:09:58.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:09:58.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:09:58.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:09:58.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:09:58.533 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:09:58.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:09:58.691 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:09:58.691 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:09:58.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:09:58.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:58.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:09:58.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836640 2026-02-22T21:09:58.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836640 2026-02-22T21:09:58.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640' 2026-02-22T21:09:58.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:58.763 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:09:58.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705821 2026-02-22T21:09:58.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705821 2026-02-22T21:09:58.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640 1-38654705821' 2026-02-22T21:09:58.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:09:58.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:09:58.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542299 2026-02-22T21:09:58.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542299 2026-02-22T21:09:58.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836640 1-38654705821 2-60129542299' 2026-02-22T21:09:58.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:09:58.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836640 2026-02-22T21:09:58.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:09:58.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:09:58.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836640 2026-02-22T21:09:58.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:09:58.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836640 2026-02-22T21:09:58.914 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836640' 2026-02-22T21:09:58.914 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836640 2026-02-22T21:09:58.914 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:09:59.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836638 -lt 21474836640 2026-02-22T21:09:59.066 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:00.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:00.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:00.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836641 -lt 21474836640 2026-02-22T21:10:00.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:00.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705821 2026-02-22T21:10:00.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:00.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:00.230 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705821 2026-02-22T21:10:00.230 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:00.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705821 2026-02-22T21:10:00.231 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705821 2026-02-22T21:10:00.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705821' 2026-02-22T21:10:00.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:00.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705822 -lt 38654705821 2026-02-22T21:10:00.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:00.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542299 2026-02-22T21:10:00.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:00.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:00.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542299 2026-02-22T21:10:00.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:00.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542299 2026-02-22T21:10:00.403 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542299 2026-02-22T21:10:00.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542299' 2026-02-22T21:10:00.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:00.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542300 -lt 60129542299 2026-02-22T21:10:00.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:00.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:00.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:00.715 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:00.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:00.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:00.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:00.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:00.725 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:00.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:00.880 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:00.880 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:00.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:00.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:00.880 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:00.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836643 2026-02-22T21:10:00.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836643 2026-02-22T21:10:00.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643' 2026-02-22T21:10:00.950 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:00.951 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:01.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705825 2026-02-22T21:10:01.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705825 2026-02-22T21:10:01.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643 1-38654705825' 2026-02-22T21:10:01.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:01.023 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:01.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542302 2026-02-22T21:10:01.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542302 2026-02-22T21:10:01.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836643 1-38654705825 2-60129542302' 2026-02-22T21:10:01.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:01.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836643 2026-02-22T21:10:01.095 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:01.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:01.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836643 2026-02-22T21:10:01.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:01.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836643 2026-02-22T21:10:01.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836643' 2026-02-22T21:10:01.099 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836643 2026-02-22T21:10:01.099 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:01.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836641 -lt 21474836643 2026-02-22T21:10:01.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:02.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:02.265 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:02.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836644 -lt 21474836643 2026-02-22T21:10:02.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:02.426 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705825 2026-02-22T21:10:02.427 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:02.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:02.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705825 2026-02-22T21:10:02.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:02.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705825 2026-02-22T21:10:02.431 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705825 2026-02-22T21:10:02.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705825' 2026-02-22T21:10:02.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:02.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705825 -lt 38654705825 2026-02-22T21:10:02.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:02.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542302 2026-02-22T21:10:02.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:02.604 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:02.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542302 2026-02-22T21:10:02.605 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:02.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542302 2026-02-22T21:10:02.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542302' 2026-02-22T21:10:02.607 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542302 2026-02-22T21:10:02.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:02.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542303 -lt 60129542302 2026-02-22T21:10:02.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:02.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:02.772 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:02.923 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:02.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:02.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:02.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:02.933 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:02.934 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:03.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:03.087 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:03.087 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:03.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:03.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:03.087 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:03.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836646 2026-02-22T21:10:03.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836646 2026-02-22T21:10:03.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646' 2026-02-22T21:10:03.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:03.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:03.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705828 2026-02-22T21:10:03.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705828 2026-02-22T21:10:03.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646 1-38654705828' 2026-02-22T21:10:03.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:03.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:03.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542305 2026-02-22T21:10:03.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542305 2026-02-22T21:10:03.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836646 1-38654705828 2-60129542305' 2026-02-22T21:10:03.305 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:03.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836646 2026-02-22T21:10:03.306 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:03.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:03.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836646 2026-02-22T21:10:03.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:03.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836646 2026-02-22T21:10:03.310 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836646 2026-02-22T21:10:03.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836646' 2026-02-22T21:10:03.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:03.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836644 -lt 21474836646 2026-02-22T21:10:03.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:04.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:04.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:04.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836647 -lt 21474836646 2026-02-22T21:10:04.633 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:04.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705828 2026-02-22T21:10:04.634 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:04.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:04.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705828 2026-02-22T21:10:04.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:04.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705828 2026-02-22T21:10:04.637 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705828 2026-02-22T21:10:04.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705828' 2026-02-22T21:10:04.638 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:04.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705828 -lt 38654705828 2026-02-22T21:10:04.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:04.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542305 2026-02-22T21:10:04.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:04.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:04.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542305 2026-02-22T21:10:04.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:04.810 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542305 2026-02-22T21:10:04.810 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542305 2026-02-22T21:10:04.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542305' 2026-02-22T21:10:04.811 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:04.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542306 -lt 60129542305 2026-02-22T21:10:04.981 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:04.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:04.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:05.127 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:05.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:05.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:05.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:05.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:05.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:05.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:05.292 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:05.292 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:05.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:05.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:05.292 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:05.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836649 2026-02-22T21:10:05.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836649 2026-02-22T21:10:05.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836649' 2026-02-22T21:10:05.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:05.365 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:05.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705831 2026-02-22T21:10:05.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705831 2026-02-22T21:10:05.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836649 1-38654705831' 2026-02-22T21:10:05.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:05.435 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:05.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542309 2026-02-22T21:10:05.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542309 2026-02-22T21:10:05.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836649 1-38654705831 2-60129542309' 2026-02-22T21:10:05.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:05.507 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836649 2026-02-22T21:10:05.507 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:05.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:05.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836649 2026-02-22T21:10:05.509 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:05.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836649 2026-02-22T21:10:05.511 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836649 2026-02-22T21:10:05.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836649' 2026-02-22T21:10:05.511 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:05.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836650 -lt 21474836649 2026-02-22T21:10:05.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:05.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705831 2026-02-22T21:10:05.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:05.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:05.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705831 2026-02-22T21:10:05.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:05.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705831 2026-02-22T21:10:05.684 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705831 2026-02-22T21:10:05.685 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705831' 2026-02-22T21:10:05.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:05.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705831 -lt 38654705831 2026-02-22T21:10:05.838 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:05.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542309 2026-02-22T21:10:05.839 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:05.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:05.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542309 2026-02-22T21:10:05.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:05.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542309 2026-02-22T21:10:05.843 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542309 2026-02-22T21:10:05.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542309' 2026-02-22T21:10:05.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:06.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542309 -lt 60129542309 2026-02-22T21:10:06.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:06.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:06.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:06.158 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:06.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:06.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:06.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:06.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:06.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:06.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:06.320 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:06.320 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:06.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:06.320 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:06.320 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:06.391 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836652 2026-02-22T21:10:06.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836652 2026-02-22T21:10:06.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836652' 2026-02-22T21:10:06.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:06.392 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:06.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705833 2026-02-22T21:10:06.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705833 2026-02-22T21:10:06.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836652 1-38654705833' 2026-02-22T21:10:06.462 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:06.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:06.533 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542311 2026-02-22T21:10:06.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542311 2026-02-22T21:10:06.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836652 1-38654705833 2-60129542311' 2026-02-22T21:10:06.534 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:06.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836652 2026-02-22T21:10:06.534 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:06.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:06.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836652 2026-02-22T21:10:06.536 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:06.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836652 2026-02-22T21:10:06.538 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836652 2026-02-22T21:10:06.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836652' 2026-02-22T21:10:06.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:06.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836650 -lt 21474836652 2026-02-22T21:10:06.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:07.702 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:07.703 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:07.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836653 -lt 21474836652 2026-02-22T21:10:07.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:07.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705833 2026-02-22T21:10:07.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:07.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:07.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705833 2026-02-22T21:10:07.862 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:07.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705833 2026-02-22T21:10:07.864 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705833 2026-02-22T21:10:07.864 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705833' 2026-02-22T21:10:07.864 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:08.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705834 -lt 38654705833 2026-02-22T21:10:08.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:08.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542311 2026-02-22T21:10:08.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:08.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:08.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542311 2026-02-22T21:10:08.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:08.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542311 2026-02-22T21:10:08.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542311' 2026-02-22T21:10:08.038 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542311 2026-02-22T21:10:08.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:08.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542312 -lt 60129542311 2026-02-22T21:10:08.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:08.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:08.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:08.345 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:08.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:08.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:08.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:08.355 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:08.355 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:08.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:08.513 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:08.513 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:08.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:08.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:08.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:08.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836655 2026-02-22T21:10:08.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836655 2026-02-22T21:10:08.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836655' 2026-02-22T21:10:08.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:08.584 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:08.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705836 2026-02-22T21:10:08.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705836 2026-02-22T21:10:08.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836655 1-38654705836' 2026-02-22T21:10:08.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:08.655 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:08.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542314 2026-02-22T21:10:08.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542314 2026-02-22T21:10:08.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836655 1-38654705836 2-60129542314' 2026-02-22T21:10:08.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:08.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836655 2026-02-22T21:10:08.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:08.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:08.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836655 2026-02-22T21:10:08.728 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:08.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836655 2026-02-22T21:10:08.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836655' 2026-02-22T21:10:08.730 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836655 2026-02-22T21:10:08.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:08.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836653 -lt 21474836655 2026-02-22T21:10:08.898 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:09.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:09.900 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:10.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836656 -lt 21474836655 2026-02-22T21:10:10.057 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:10.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705836 2026-02-22T21:10:10.058 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:10.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:10.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705836 2026-02-22T21:10:10.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:10.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705836 2026-02-22T21:10:10.062 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705836 2026-02-22T21:10:10.062 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705836' 2026-02-22T21:10:10.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:10.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705837 -lt 38654705836 2026-02-22T21:10:10.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:10.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542314 2026-02-22T21:10:10.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:10.224 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:10.224 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542314 2026-02-22T21:10:10.225 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:10.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542314 2026-02-22T21:10:10.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542314' 2026-02-22T21:10:10.226 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542314 2026-02-22T21:10:10.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:10.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542315 -lt 60129542314 2026-02-22T21:10:10.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:10.395 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:10.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:10.542 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:10.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:10.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:10.551 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:10.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:10.552 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:10.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:10.705 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:10.705 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:10.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:10.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:10.705 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:10.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836658 2026-02-22T21:10:10.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836658 2026-02-22T21:10:10.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658' 2026-02-22T21:10:10.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:10.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:10.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705839 2026-02-22T21:10:10.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705839 2026-02-22T21:10:10.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658 1-38654705839' 2026-02-22T21:10:10.850 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:10.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:10.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542317 2026-02-22T21:10:10.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542317 2026-02-22T21:10:10.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836658 1-38654705839 2-60129542317' 2026-02-22T21:10:10.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:10.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836658 2026-02-22T21:10:10.923 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:10.923 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:10.924 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836658 2026-02-22T21:10:10.924 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:10.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836658 2026-02-22T21:10:10.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836658' 2026-02-22T21:10:10.926 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836658 2026-02-22T21:10:10.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:11.088 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836656 -lt 21474836658 2026-02-22T21:10:11.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:12.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:12.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:12.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836659 -lt 21474836658 2026-02-22T21:10:12.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:12.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705839 2026-02-22T21:10:12.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:12.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:12.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705839 2026-02-22T21:10:12.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:12.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705839 2026-02-22T21:10:12.257 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705839 2026-02-22T21:10:12.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705839' 2026-02-22T21:10:12.258 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:12.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705840 -lt 38654705839 2026-02-22T21:10:12.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:12.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542317 2026-02-22T21:10:12.413 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:12.414 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:12.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542317 2026-02-22T21:10:12.415 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:12.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542317 2026-02-22T21:10:12.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542317' 2026-02-22T21:10:12.417 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542317 2026-02-22T21:10:12.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:12.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542318 -lt 60129542317 2026-02-22T21:10:12.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:12.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:12.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:12.725 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:12.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:12.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:12.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:12.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:12.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:12.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:12.891 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:12.892 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:12.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:12.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:12.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:12.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836661 2026-02-22T21:10:12.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836661 2026-02-22T21:10:12.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661' 2026-02-22T21:10:12.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:12.964 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:13.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705843 2026-02-22T21:10:13.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705843 2026-02-22T21:10:13.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661 1-38654705843' 2026-02-22T21:10:13.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:13.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:13.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542320 2026-02-22T21:10:13.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542320 2026-02-22T21:10:13.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836661 1-38654705843 2-60129542320' 2026-02-22T21:10:13.110 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:13.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836661 2026-02-22T21:10:13.111 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:13.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:13.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836661 2026-02-22T21:10:13.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:13.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836661 2026-02-22T21:10:13.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836661' 2026-02-22T21:10:13.114 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836661 2026-02-22T21:10:13.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:13.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836659 -lt 21474836661 2026-02-22T21:10:13.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:14.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:14.280 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:14.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836662 -lt 21474836661 2026-02-22T21:10:14.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:14.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705843 2026-02-22T21:10:14.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:14.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:14.444 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705843 2026-02-22T21:10:14.444 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:14.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705843 2026-02-22T21:10:14.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705843' 2026-02-22T21:10:14.446 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705843 2026-02-22T21:10:14.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:14.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705843 -lt 38654705843 2026-02-22T21:10:14.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:14.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542320 2026-02-22T21:10:14.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:14.612 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:14.613 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542320 2026-02-22T21:10:14.613 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:14.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542320 2026-02-22T21:10:14.615 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542320 2026-02-22T21:10:14.615 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542320' 2026-02-22T21:10:14.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:14.783 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542321 -lt 60129542320 2026-02-22T21:10:14.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:14.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:14.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:14.929 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:14.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:14.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:14.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:14.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:14.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:15.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:15.097 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:15.097 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:15.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:15.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:15.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:15.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836664 2026-02-22T21:10:15.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836664 2026-02-22T21:10:15.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664' 2026-02-22T21:10:15.168 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:15.168 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:15.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705846 2026-02-22T21:10:15.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705846 2026-02-22T21:10:15.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664 1-38654705846' 2026-02-22T21:10:15.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:15.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:15.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542323 2026-02-22T21:10:15.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542323 2026-02-22T21:10:15.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836664 1-38654705846 2-60129542323' 2026-02-22T21:10:15.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:15.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836664 2026-02-22T21:10:15.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:15.313 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:15.313 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836664 2026-02-22T21:10:15.314 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:15.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836664 2026-02-22T21:10:15.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836664' 2026-02-22T21:10:15.315 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836664 2026-02-22T21:10:15.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:15.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836662 -lt 21474836664 2026-02-22T21:10:15.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:16.480 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:16.481 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:16.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836665 -lt 21474836664 2026-02-22T21:10:16.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:16.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705846 2026-02-22T21:10:16.641 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:16.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:16.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705846 2026-02-22T21:10:16.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:16.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705846 2026-02-22T21:10:16.645 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705846 2026-02-22T21:10:16.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705846' 2026-02-22T21:10:16.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:16.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705846 -lt 38654705846 2026-02-22T21:10:16.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:16.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542323 2026-02-22T21:10:16.802 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:16.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:16.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542323 2026-02-22T21:10:16.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:16.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542323 2026-02-22T21:10:16.806 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542323 2026-02-22T21:10:16.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542323' 2026-02-22T21:10:16.806 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:16.963 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542324 -lt 60129542323 2026-02-22T21:10:16.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:16.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:16.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:17.109 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:17.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:17.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:17.119 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:17.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:17.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:17.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:17.271 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:17.271 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:17.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:17.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:17.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:17.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836667 2026-02-22T21:10:17.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836667 2026-02-22T21:10:17.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836667' 2026-02-22T21:10:17.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:17.345 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:17.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705849 2026-02-22T21:10:17.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705849 2026-02-22T21:10:17.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836667 1-38654705849' 2026-02-22T21:10:17.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:17.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:17.489 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542327 2026-02-22T21:10:17.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542327 2026-02-22T21:10:17.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836667 1-38654705849 2-60129542327' 2026-02-22T21:10:17.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:17.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836667 2026-02-22T21:10:17.491 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:17.491 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:17.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836667 2026-02-22T21:10:17.492 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:17.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836667 2026-02-22T21:10:17.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836667' 2026-02-22T21:10:17.494 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836667 2026-02-22T21:10:17.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:17.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836668 -lt 21474836667 2026-02-22T21:10:17.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:17.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705849 2026-02-22T21:10:17.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:17.664 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:17.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705849 2026-02-22T21:10:17.665 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:17.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705849 2026-02-22T21:10:17.667 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705849 2026-02-22T21:10:17.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705849' 2026-02-22T21:10:17.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:17.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705849 -lt 38654705849 2026-02-22T21:10:17.839 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:17.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542327 2026-02-22T21:10:17.840 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:17.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:17.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542327 2026-02-22T21:10:17.841 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:17.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542327 2026-02-22T21:10:17.843 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542327 2026-02-22T21:10:17.843 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542327' 2026-02-22T21:10:17.844 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:18.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542327 -lt 60129542327 2026-02-22T21:10:18.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:18.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:18.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:18.158 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:18.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:18.169 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:18.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:18.170 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:18.170 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:18.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:18.328 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:18.328 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:18.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:18.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:18.328 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836670 2026-02-22T21:10:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836670 2026-02-22T21:10:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836670' 2026-02-22T21:10:18.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:18.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:18.472 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705851 2026-02-22T21:10:18.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705851 2026-02-22T21:10:18.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836670 1-38654705851' 2026-02-22T21:10:18.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:18.473 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:18.548 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542329 2026-02-22T21:10:18.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542329 2026-02-22T21:10:18.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836670 1-38654705851 2-60129542329' 2026-02-22T21:10:18.549 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:18.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836670 2026-02-22T21:10:18.549 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:18.550 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:18.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836670 2026-02-22T21:10:18.551 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:18.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836670 2026-02-22T21:10:18.553 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836670 2026-02-22T21:10:18.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836670' 2026-02-22T21:10:18.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:18.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836668 -lt 21474836670 2026-02-22T21:10:18.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:19.711 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:19.711 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:19.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836671 -lt 21474836670 2026-02-22T21:10:19.867 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:19.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705851 2026-02-22T21:10:19.868 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:19.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:19.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705851 2026-02-22T21:10:19.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:19.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705851 2026-02-22T21:10:19.872 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705851 2026-02-22T21:10:19.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705851' 2026-02-22T21:10:19.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:20.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705852 -lt 38654705851 2026-02-22T21:10:20.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:20.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542329 2026-02-22T21:10:20.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:20.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:20.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542329 2026-02-22T21:10:20.044 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:20.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542329 2026-02-22T21:10:20.046 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542329 2026-02-22T21:10:20.046 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542329' 2026-02-22T21:10:20.046 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:20.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542330 -lt 60129542329 2026-02-22T21:10:20.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:20.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:20.212 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:20.361 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:20.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:20.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:20.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:20.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:20.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:20.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:20.525 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:20.525 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:20.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:20.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:20.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:20.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836673 2026-02-22T21:10:20.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836673 2026-02-22T21:10:20.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836673' 2026-02-22T21:10:20.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:20.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:20.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705854 2026-02-22T21:10:20.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705854 2026-02-22T21:10:20.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836673 1-38654705854' 2026-02-22T21:10:20.671 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:20.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:20.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542332 2026-02-22T21:10:20.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542332 2026-02-22T21:10:20.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836673 1-38654705854 2-60129542332' 2026-02-22T21:10:20.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:20.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836673 2026-02-22T21:10:20.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:20.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:20.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836673 2026-02-22T21:10:20.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:20.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836673 2026-02-22T21:10:20.747 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836673' 2026-02-22T21:10:20.747 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836673 2026-02-22T21:10:20.748 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:20.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836671 -lt 21474836673 2026-02-22T21:10:20.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:21.917 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:21.917 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:22.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836674 -lt 21474836673 2026-02-22T21:10:22.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:22.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705854 2026-02-22T21:10:22.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:22.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:22.076 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705854 2026-02-22T21:10:22.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:22.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705854 2026-02-22T21:10:22.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705854' 2026-02-22T21:10:22.078 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705854 2026-02-22T21:10:22.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:22.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705855 -lt 38654705854 2026-02-22T21:10:22.244 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:22.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542332 2026-02-22T21:10:22.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:22.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:22.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542332 2026-02-22T21:10:22.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:22.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542332 2026-02-22T21:10:22.249 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542332' 2026-02-22T21:10:22.249 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542332 2026-02-22T21:10:22.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:22.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542333 -lt 60129542332 2026-02-22T21:10:22.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:22.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:22.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:22.553 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:22.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:22.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:22.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:22.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:22.564 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:22.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:22.723 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:22.723 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:22.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:22.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:22.723 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:22.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836676 2026-02-22T21:10:22.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836676 2026-02-22T21:10:22.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836676' 2026-02-22T21:10:22.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:22.797 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:22.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705857 2026-02-22T21:10:22.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705857 2026-02-22T21:10:22.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836676 1-38654705857' 2026-02-22T21:10:22.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:22.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:22.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542335 2026-02-22T21:10:22.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542335 2026-02-22T21:10:22.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836676 1-38654705857 2-60129542335' 2026-02-22T21:10:22.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:22.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836676 2026-02-22T21:10:22.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:22.940 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:22.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836676 2026-02-22T21:10:22.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:22.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836676 2026-02-22T21:10:22.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836676' 2026-02-22T21:10:22.943 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836676 2026-02-22T21:10:22.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:23.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836674 -lt 21474836676 2026-02-22T21:10:23.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:24.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:24.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:24.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836677 -lt 21474836676 2026-02-22T21:10:24.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:24.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705857 2026-02-22T21:10:24.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:24.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:24.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705857 2026-02-22T21:10:24.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:24.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705857 2026-02-22T21:10:24.270 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705857 2026-02-22T21:10:24.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705857' 2026-02-22T21:10:24.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:24.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705858 -lt 38654705857 2026-02-22T21:10:24.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:24.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542335 2026-02-22T21:10:24.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:24.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:24.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542335 2026-02-22T21:10:24.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:24.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542335 2026-02-22T21:10:24.444 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542335 2026-02-22T21:10:24.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542335' 2026-02-22T21:10:24.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:24.609 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542336 -lt 60129542335 2026-02-22T21:10:24.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:24.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:24.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:24.761 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:24.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:24.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:24.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:24.771 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:24.772 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:24.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:24.927 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:24.928 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:24.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:24.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:24.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:24.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836679 2026-02-22T21:10:24.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836679 2026-02-22T21:10:24.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836679' 2026-02-22T21:10:24.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:24.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:25.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705861 2026-02-22T21:10:25.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705861 2026-02-22T21:10:25.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836679 1-38654705861' 2026-02-22T21:10:25.072 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:25.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:25.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542338 2026-02-22T21:10:25.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542338 2026-02-22T21:10:25.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836679 1-38654705861 2-60129542338' 2026-02-22T21:10:25.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:25.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836679 2026-02-22T21:10:25.145 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:25.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:25.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836679 2026-02-22T21:10:25.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:25.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836679 2026-02-22T21:10:25.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836679' 2026-02-22T21:10:25.148 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836679 2026-02-22T21:10:25.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:25.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836677 -lt 21474836679 2026-02-22T21:10:25.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:26.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:26.315 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:26.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836680 -lt 21474836679 2026-02-22T21:10:26.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:26.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705861 2026-02-22T21:10:26.474 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:26.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:26.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705861 2026-02-22T21:10:26.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:26.477 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705861 2026-02-22T21:10:26.478 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705861 2026-02-22T21:10:26.478 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705861' 2026-02-22T21:10:26.478 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:26.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705861 -lt 38654705861 2026-02-22T21:10:26.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:26.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542338 2026-02-22T21:10:26.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:26.646 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:26.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542338 2026-02-22T21:10:26.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:26.648 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542338 2026-02-22T21:10:26.649 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542338 2026-02-22T21:10:26.649 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542338' 2026-02-22T21:10:26.649 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:26.807 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542339 -lt 60129542338 2026-02-22T21:10:26.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:26.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:26.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:26.955 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:26.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:26.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:26.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:26.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:26.965 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:27.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:27.124 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:27.124 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:27.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:27.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:27.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:27.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836682 2026-02-22T21:10:27.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836682 2026-02-22T21:10:27.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836682' 2026-02-22T21:10:27.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:27.194 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:27.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705864 2026-02-22T21:10:27.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705864 2026-02-22T21:10:27.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836682 1-38654705864' 2026-02-22T21:10:27.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:27.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:27.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542341 2026-02-22T21:10:27.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542341 2026-02-22T21:10:27.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836682 1-38654705864 2-60129542341' 2026-02-22T21:10:27.338 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:27.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836682 2026-02-22T21:10:27.339 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:27.340 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:27.341 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836682 2026-02-22T21:10:27.341 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:27.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836682 2026-02-22T21:10:27.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836682' 2026-02-22T21:10:27.342 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836682 2026-02-22T21:10:27.343 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:27.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836680 -lt 21474836682 2026-02-22T21:10:27.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:28.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:28.507 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:28.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836683 -lt 21474836682 2026-02-22T21:10:28.666 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:28.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705864 2026-02-22T21:10:28.667 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:28.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:28.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705864 2026-02-22T21:10:28.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:28.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705864 2026-02-22T21:10:28.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705864' 2026-02-22T21:10:28.670 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705864 2026-02-22T21:10:28.671 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:28.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705864 -lt 38654705864 2026-02-22T21:10:28.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:28.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542341 2026-02-22T21:10:28.831 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:28.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:28.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542341 2026-02-22T21:10:28.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:28.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542341 2026-02-22T21:10:28.834 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542341 2026-02-22T21:10:28.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542341' 2026-02-22T21:10:28.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:29.003 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542342 -lt 60129542341 2026-02-22T21:10:29.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:29.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:29.004 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:29.149 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:29.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:29.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:29.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:29.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:29.160 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:29.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:29.311 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:29.311 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:29.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:29.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:29.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:29.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836685 2026-02-22T21:10:29.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836685 2026-02-22T21:10:29.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836685' 2026-02-22T21:10:29.383 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:29.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:29.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705867 2026-02-22T21:10:29.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705867 2026-02-22T21:10:29.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836685 1-38654705867' 2026-02-22T21:10:29.454 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:29.454 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:29.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542345 2026-02-22T21:10:29.522 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542345 2026-02-22T21:10:29.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836685 1-38654705867 2-60129542345' 2026-02-22T21:10:29.523 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:29.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836685 2026-02-22T21:10:29.523 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:29.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:29.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836685 2026-02-22T21:10:29.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:29.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836685 2026-02-22T21:10:29.527 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836685 2026-02-22T21:10:29.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836685' 2026-02-22T21:10:29.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:29.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836686 -lt 21474836685 2026-02-22T21:10:29.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:29.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705867 2026-02-22T21:10:29.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:29.692 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:29.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705867 2026-02-22T21:10:29.693 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:29.694 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705867 2026-02-22T21:10:29.694 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705867 2026-02-22T21:10:29.695 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705867' 2026-02-22T21:10:29.695 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:29.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705867 -lt 38654705867 2026-02-22T21:10:29.853 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:29.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542345 2026-02-22T21:10:29.854 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:29.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:29.855 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542345 2026-02-22T21:10:29.856 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:29.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542345 2026-02-22T21:10:29.857 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542345' 2026-02-22T21:10:29.857 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542345 2026-02-22T21:10:29.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:30.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542344 -lt 60129542345 2026-02-22T21:10:30.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:31.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:31.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:31.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542344 -lt 60129542345 2026-02-22T21:10:31.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:32.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-22T21:10:32.187 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:32.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542347 -lt 60129542345 2026-02-22T21:10:32.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:32.345 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:32.346 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:32.500 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:32.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:32.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:32.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:32.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:32.510 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:32.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:32.669 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:32.669 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:32.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:32.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:32.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:32.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836690 2026-02-22T21:10:32.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836690 2026-02-22T21:10:32.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836690' 2026-02-22T21:10:32.743 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:32.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:32.814 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705871 2026-02-22T21:10:32.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705871 2026-02-22T21:10:32.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836690 1-38654705871' 2026-02-22T21:10:32.815 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:32.815 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:32.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542349 2026-02-22T21:10:32.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542349 2026-02-22T21:10:32.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836690 1-38654705871 2-60129542349' 2026-02-22T21:10:32.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:32.886 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836690 2026-02-22T21:10:32.886 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:32.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:32.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836690 2026-02-22T21:10:32.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:32.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836690 2026-02-22T21:10:32.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836690' 2026-02-22T21:10:32.891 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836690 2026-02-22T21:10:32.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:33.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836688 -lt 21474836690 2026-02-22T21:10:33.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:34.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:34.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:34.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836691 -lt 21474836690 2026-02-22T21:10:34.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:34.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705871 2026-02-22T21:10:34.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:34.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:34.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705871 2026-02-22T21:10:34.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:34.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705871 2026-02-22T21:10:34.211 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705871 2026-02-22T21:10:34.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705871' 2026-02-22T21:10:34.211 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:34.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705872 -lt 38654705871 2026-02-22T21:10:34.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:34.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542349 2026-02-22T21:10:34.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:34.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:34.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542349 2026-02-22T21:10:34.383 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:34.384 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542349 2026-02-22T21:10:34.385 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542349' 2026-02-22T21:10:34.385 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542349 2026-02-22T21:10:34.385 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:34.545 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542350 -lt 60129542349 2026-02-22T21:10:34.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:34.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:34.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:34.690 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:34.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:34.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:34.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:34.701 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:34.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:34.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:34.858 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:34.858 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:34.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:34.858 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:34.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:34.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836693 2026-02-22T21:10:34.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836693 2026-02-22T21:10:34.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836693' 2026-02-22T21:10:34.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:34.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:35.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705875 2026-02-22T21:10:35.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705875 2026-02-22T21:10:35.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836693 1-38654705875' 2026-02-22T21:10:35.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:35.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:35.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542352 2026-02-22T21:10:35.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542352 2026-02-22T21:10:35.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836693 1-38654705875 2-60129542352' 2026-02-22T21:10:35.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:35.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836693 2026-02-22T21:10:35.075 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:35.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:35.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836693 2026-02-22T21:10:35.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:35.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836693 2026-02-22T21:10:35.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836693' 2026-02-22T21:10:35.079 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836693 2026-02-22T21:10:35.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:35.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836691 -lt 21474836693 2026-02-22T21:10:35.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:36.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:36.243 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:36.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836694 -lt 21474836693 2026-02-22T21:10:36.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:36.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705875 2026-02-22T21:10:36.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:36.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:36.409 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705875 2026-02-22T21:10:36.409 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:36.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705875 2026-02-22T21:10:36.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705875' 2026-02-22T21:10:36.411 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705875 2026-02-22T21:10:36.411 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:36.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705875 -lt 38654705875 2026-02-22T21:10:36.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:36.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542352 2026-02-22T21:10:36.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:36.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:36.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542352 2026-02-22T21:10:36.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:36.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542352 2026-02-22T21:10:36.573 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542352 2026-02-22T21:10:36.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542352' 2026-02-22T21:10:36.573 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:36.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542353 -lt 60129542352 2026-02-22T21:10:36.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:36.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:36.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:36.876 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:36.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:375: _scrub_abort: continue 2026-02-22T21:10:36.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:370: _scrub_abort: for i in $(seq 0 200) 2026-02-22T21:10:36.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:372: _scrub_abort: flush_pg_stats 2026-02-22T21:10:36.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:10:36.888 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:10:37.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:10:37.045 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:10:37.045 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:10:37.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:10:37.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:37.045 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:10:37.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836696 2026-02-22T21:10:37.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836696 2026-02-22T21:10:37.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836696' 2026-02-22T21:10:37.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:37.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:10:37.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705878 2026-02-22T21:10:37.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705878 2026-02-22T21:10:37.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836696 1-38654705878' 2026-02-22T21:10:37.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:10:37.190 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:10:37.263 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542355 2026-02-22T21:10:37.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542355 2026-02-22T21:10:37.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836696 1-38654705878 2-60129542355' 2026-02-22T21:10:37.264 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:37.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836696 2026-02-22T21:10:37.264 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:37.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:10:37.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836696 2026-02-22T21:10:37.266 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:37.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836696 2026-02-22T21:10:37.268 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836696' 2026-02-22T21:10:37.268 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836696 2026-02-22T21:10:37.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:37.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836694 -lt 21474836696 2026-02-22T21:10:37.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:10:38.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:10:38.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:10:38.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836697 -lt 21474836696 2026-02-22T21:10:38.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:38.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705878 2026-02-22T21:10:38.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:38.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:10:38.597 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-38654705878 2026-02-22T21:10:38.597 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:38.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=38654705878 2026-02-22T21:10:38.599 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 38654705878 2026-02-22T21:10:38.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 38654705878' 2026-02-22T21:10:38.599 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:10:38.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 38654705878 -lt 38654705878 2026-02-22T21:10:38.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:10:38.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542355 2026-02-22T21:10:38.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:10:38.767 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:10:38.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-60129542355 2026-02-22T21:10:38.768 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:10:38.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=60129542355 2026-02-22T21:10:38.770 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 60129542355 2026-02-22T21:10:38.770 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 60129542355' 2026-02-22T21:10:38.770 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:10:38.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 60129542356 -lt 60129542355 2026-02-22T21:10:38.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: ceph pg dump pgs 2026-02-22T21:10:38.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep '^1.0' 2026-02-22T21:10:38.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:373: _scrub_abort: grep -q scrubbing 2026-02-22T21:10:39.076 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:10:39.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:378: _scrub_abort: break 2026-02-22T21:10:39.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:380: _scrub_abort: set +o pipefail 2026-02-22T21:10:39.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:382: _scrub_abort: sleep 5 2026-02-22T21:10:44.087 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:384: _scrub_abort: grep 'noscrub set, aborting' td/osd-scrub-test/osd.1.log 2026-02-22T21:10:44.096 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:08:40.643+0000 7fbfda98a640 10 osd.1 pg_epoch: 18 pg[1.0( v 17'1000 (0'0,17'1000] local-lis/les=15/16 n=1000 ec=15/15 lis/c=15/15 les/c/f=16/16/0 sis=15) [1,0,2] r=0 lpr=15 crt=17'1000 lcod 17'999 mlcod 17'999 active+clean+scrubbing(periodic-regular)] scrubber: noscrub set, aborting 2026-02-22T21:10:44.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:390: _scrub_abort: get_last_scrub_stamp 1.0 2026-02-22T21:10:44.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:44.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:44.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:44.098 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:44.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:390: _scrub_abort: local last_scrub=2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:44.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:391: _scrub_abort: ceph config set osd osd_scrub_sleep 0.1 2026-02-22T21:10:44.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:393: _scrub_abort: ceph osd unset noscrub 2026-02-22T21:10:44.608 INFO:tasks.workunit.client.0.trial150.stderr:noscrub is unset 2026-02-22T21:10:44.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:394: _scrub_abort: '[' scrub = deep-scrub ']' 2026-02-22T21:10:44.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:398: _scrub_abort: TIMEOUT=500 2026-02-22T21:10:44.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:399: _scrub_abort: wait_for_scrub 1.0 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:44.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T21:10:44.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:44.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T21:10:44.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T21:10:44.622 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:44.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:44.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:44.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:44.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:44.623 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:44.784 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:44.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:45.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:45.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:45.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:45.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:45.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:45.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:45.786 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:45.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:45.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:46.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:46.947 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:46.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:46.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:46.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:46.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:46.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:47.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:47.104 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:48.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:48.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:48.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:48.105 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:48.106 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:48.106 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:48.106 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:48.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:48.265 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:49.266 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:49.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:49.267 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:49.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:49.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:49.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:49.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:49.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:49.420 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:50.421 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:50.422 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:50.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:50.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:50.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:50.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:50.422 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:50.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:50.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:51.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:51.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:51.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:51.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:51.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:51.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:51.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:51.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:51.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:52.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:52.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:52.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:52.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:52.739 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:52.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:52.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:52.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:52.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:53.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:53.900 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:53.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:53.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:53.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:53.901 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:53.902 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:54.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:54.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:55.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:55.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:55.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:55.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:55.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:55.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:55.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:55.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:55.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:56.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:56.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:56.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:56.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:56.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:56.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:56.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:56.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:56.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:57.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:57.380 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:57.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:57.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:57.380 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:57.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:57.381 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:57.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:57.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:58.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:58.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:58.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:58.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:58.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:58.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:58.539 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:58.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:58.696 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:10:59.697 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:10:59.698 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:10:59.698 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:10:59.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:10:59.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:10:59.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:10:59.699 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:10:59.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:10:59.855 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:00.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:00.856 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:00.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:00.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:00.857 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:00.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:00.858 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:01.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:01.013 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:02.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:02.014 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:02.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:02.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:02.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:02.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:02.015 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:02.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:02.171 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:03.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:03.172 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:03.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:03.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:03.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:03.173 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:03.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:03.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:03.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:04.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:04.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:04.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:04.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:04.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:04.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:04.330 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:04.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:04.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:05.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:05.486 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:05.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:05.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:05.487 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:05.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:05.488 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:05.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:05.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:06.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:06.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:06.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:06.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:06.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:06.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:06.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:06.796 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:06.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:07.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:07.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:07.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:07.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:07.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:07.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:07.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:07.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:07.956 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:08.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:08.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:08.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:08.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:08.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:08.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:08.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:09.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:09.117 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:10.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:10.118 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:10.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:10.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:10.118 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:10.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:10.119 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:10.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:10.279 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:11.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:11.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:11.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:11.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:11.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:11.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:11.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:11.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:11.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:12.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:12.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:12.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:12.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:12.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:12.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:12.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:12.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:12.598 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:13.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:13.599 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:13.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:13.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:13.600 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:13.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:13.601 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:13.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:13.757 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:14.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:14.758 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:14.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:14.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:14.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:14.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:14.759 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:14.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:14.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:15.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:15.920 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:15.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:15.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:15.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:15.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:15.922 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:16.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:16.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:17.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:17.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:17.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:17.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:17.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:17.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:17.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:17.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:17.237 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:18.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:18.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:18.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:18.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:18.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:18.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:18.240 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:18.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:18.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:19.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:19.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:19.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:19.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:19.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:19.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:19.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:19.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:19.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:20.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:20.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:20.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:20.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:20.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:20.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:20.559 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:20.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:20.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:21.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:21.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:21.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:21.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:21.718 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:21.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:21.719 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:21.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:21.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:22.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:22.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:22.878 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:22.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:22.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:22.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:22.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:23.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:23.035 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:24.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:24.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:24.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:24.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:24.036 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:24.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:24.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:24.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:24.194 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:25.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:25.195 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:25.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:25.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:25.196 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:25.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:25.197 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:25.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:25.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:26.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:26.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:26.353 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:26.353 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:26.353 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:26.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:26.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:26.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:26.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:27.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:27.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:27.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:27.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:27.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:27.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:27.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:27.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:27.672 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:28.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:28.673 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:28.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:28.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:28.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:28.674 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:28.675 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:28.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:28.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:29.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:29.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:29.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:29.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:29.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:29.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:29.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:29.992 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:29.993 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:30.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:30.994 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:30.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:30.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:30.995 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:30.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:30.996 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:31.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:31.152 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:32.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:32.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:32.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:32.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:32.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:32.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:32.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:32.314 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:32.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:33.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:33.316 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:33.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:33.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:33.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:33.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:33.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:33.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:33.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:34.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:34.476 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:34.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:34.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:34.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:34.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:34.477 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:34.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:34.640 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:35.641 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:35.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:35.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:35.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:35.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:35.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:35.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:35.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:35.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:36.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:36.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:36.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:36.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:36.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:36.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:36.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:36.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:36.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:37.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:37.959 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:37.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:37.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:37.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:37.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:37.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:38.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:38.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:39.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:39.114 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:39.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:39.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:39.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:39.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:39.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:39.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:39.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:40.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:40.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:40.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:40.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:40.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:40.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:40.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:40.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:40.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:41.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:41.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:41.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:41.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:41.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:41.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:41.432 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:41.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:41.590 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:42.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:42.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:42.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:42.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:42.592 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:42.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:42.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:42.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:42.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:43.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:43.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:43.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:43.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:43.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:43.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:43.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:43.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:43.915 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:44.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:44.916 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:44.917 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:44.917 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:44.917 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:44.917 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:44.917 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:45.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:45.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:46.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:46.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:46.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:46.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:46.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:46.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:46.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:46.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:46.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:47.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:47.247 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:47.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:47.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:47.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:47.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:47.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:47.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:47.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:48.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:48.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:48.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:48.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:48.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:48.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:48.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:48.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:48.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:49.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:49.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:49.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:49.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:49.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:49.570 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:49.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:49.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:49.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:50.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:50.728 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:50.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:50.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:50.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:50.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:50.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:50.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:50.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:51.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:51.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:51.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:51.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:51.891 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:51.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:51.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:52.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:52.049 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:53.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:53.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:53.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:53.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:53.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:53.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:53.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:53.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:53.205 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:54.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:54.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:54.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:54.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:54.208 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:54.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:54.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:54.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:54.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:55.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:55.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:55.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:55.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:55.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:55.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:55.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:55.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:55.524 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:56.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:56.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:56.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:56.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:56.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:56.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:56.526 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:56.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:56.681 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:57.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:57.682 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:57.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:57.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:57.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:57.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:57.683 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:57.840 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:57.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:11:58.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:11:58.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:11:58.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:11:58.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:11:58.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:11:58.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:11:58.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:11:59.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:11:59.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:00.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:00.002 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:00.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:00.002 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:00.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:00.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:00.003 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:00.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:00.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:01.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:01.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:01.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:01.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:01.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:01.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:01.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:01.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:01.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:02.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:02.315 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:02.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:02.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:02.316 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:02.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:02.317 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:02.473 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:02.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:03.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:03.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:03.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:03.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:03.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:03.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:03.476 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:03.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:03.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:04.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:04.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:04.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:04.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:04.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:04.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:04.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:04.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:04.797 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:05.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:05.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:05.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:05.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:05.799 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:05.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:05.800 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:05.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:05.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:06.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:06.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:06.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:06.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:06.959 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:06.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:06.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:07.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:07.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:08.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:08.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:08.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:08.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:08.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:08.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:08.117 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:08.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:08.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:09.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:09.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:09.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:09.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:09.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:09.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:09.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:09.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:09.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:10.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:10.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:10.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:10.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:10.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:10.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:10.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:10.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:10.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:11.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:11.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:11.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:11.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:11.589 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:11.590 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:11.590 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:11.744 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:11.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:12.745 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:12.746 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:12.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:12.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:12.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:12.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:12.747 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:12.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:12.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:13.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:13.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:13.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:13.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:13.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:13.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:13.904 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:14.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:14.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:15.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:15.055 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:15.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:15.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:15.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:15.056 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:15.057 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:15.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:15.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:16.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:16.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:16.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:16.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:16.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:16.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:16.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:16.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:16.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:17.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:17.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:17.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:17.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:17.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:17.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:17.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:17.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:17.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:18.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:18.527 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:18.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:18.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:18.527 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:18.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:18.528 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:18.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-20T21:08:32.513200+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:18.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:12:19.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:12:19.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:12:19.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:12:19.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:19.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:19.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:19.685 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:19.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T21:12:18.457275+0000 '>' 2026-02-20T21:08:32.513200+0000 2026-02-22T21:12:19.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T21:12:19.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:400: _scrub_abort: perf_counters td/osd-scrub-test 3 2026-02-22T21:12:19.845 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-22T21:12:19.846 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-22T21:12:19.846 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-22T21:12:19.846 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-22T21:12:19.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:12:19.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-22T21:12:19.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:12:19.926 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:12:19.927 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:12:19.927 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:12:19.927 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:12:19.927 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:12:19.927 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:12:19.927 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:12:19.927 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:12:19.927 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:12:19.927 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:12:19.927 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:12:19.927 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:12:19.928 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:12:19.929 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:12:19.930 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:12:19.931 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:12:19.931 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:12:19.931 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:12:19.931 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:12:19.931 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:12:19.931 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:12:19.931 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:12:19.931 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.931 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.931 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:12:19.931 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:12:19.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:12:19.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-22T21:12:19.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:12:19.995 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:12:19.995 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:12:19.995 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:12:19.995 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:12:19.995 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:12:19.995 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:12:19.996 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:12:19.997 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:12:19.997 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:12:19.997 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:12:19.997 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:12:19.997 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:12:19.997 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:12:19.997 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:12:19.997 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:12:19.997 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:12:19.997 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.997 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.997 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:12:19.998 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 11, 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:12:19.999 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:12:20.000 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:12:20.000 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:20.000 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:20.000 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:12:20.000 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:12:20.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:12:20.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-22T21:12:20.001 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:12:20.067 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:12:20.067 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:12:20.068 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:12:20.068 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:12:20.068 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:12:20.068 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:12:20.068 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:12:20.068 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:12:20.068 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:12:20.068 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:12:20.068 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:12:20.068 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:12:20.068 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:12:20.069 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:20.069 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:20.069 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:12:20.069 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:12:20.069 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:12:20.069 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:12:20.069 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:12:20.069 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:12:20.069 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:12:20.069 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:12:20.069 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:12:20.069 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:12:20.070 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:12:20.071 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:12:20.072 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:12:20.072 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:12:20.072 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:12:20.072 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:12:20.072 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:12:20.072 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:12:20.072 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:12:20.072 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:20.072 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:12:20.072 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:12:20.072 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:12:20.072 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Teardown Test TEST_scrub_abort ------------------ 2026-02-22T21:12:20.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_scrub_abort ------------------' 2026-02-22T21:12:20.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-22T21:12:20.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:12:20.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:12:20.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:12:20.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:12:20.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:12:20.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:12:20.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:12:20.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:12:20.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:12:20.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:12:20.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:12:20.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:12:20.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:12:20.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:12:20.181 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:12:20.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:12:20.182 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:12:20.183 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:12:20.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:12:20.184 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:12:20.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:12:20.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:12:20.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:12:20.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:12:20.231 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:12:20.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:12:20.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:12:20.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:12:20.232 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Complete Test TEST_scrub_abort ------------------ 2026-02-22T21:12:20.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_scrub_abort ------------------' 2026-02-22T21:12:20.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-22T21:12:20.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_scrub_permit_time -------------------' 2026-02-22T21:12:20.233 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Prepare Test TEST_scrub_permit_time ------------------- 2026-02-22T21:12:20.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-22T21:12:20.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-22T21:12:20.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-22T21:12:20.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:12:20.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:12:20.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:12:20.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:12:20.235 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:12:20.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:12:20.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:12:20.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:12:20.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:12:20.236 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:12:20.236 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:12:20.237 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:12:20.238 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:12:20.239 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:12:20.239 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:12:20.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:12:20.240 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:12:20.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:12:20.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:12:20.242 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:12:20.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:12:20.243 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:12:20.244 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:12:20.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:12:20.245 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:12:20.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:12:20.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:12:20.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:12:20.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-22T21:12:20.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T21:12:20.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:12:20.248 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:12:20.248 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.516403 2026-02-22T21:12:20.249 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T21:12:20.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:12:20.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T21:12:20.250 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-22T21:12:20.250 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Run Test TEST_scrub_permit_time ----------------------- 2026-02-22T21:12:20.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_scrub_permit_time -----------------------' 2026-02-22T21:12:20.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_scrub_permit_time td/osd-scrub-test 2026-02-22T21:12:20.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:414: TEST_scrub_permit_time: local dir=td/osd-scrub-test 2026-02-22T21:12:20.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:415: TEST_scrub_permit_time: local poolname=test 2026-02-22T21:12:20.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:416: TEST_scrub_permit_time: local OSDS=3 2026-02-22T21:12:20.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:417: TEST_scrub_permit_time: local objects=15 2026-02-22T21:12:20.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:419: TEST_scrub_permit_time: TESTDATA=testdata.516403 2026-02-22T21:12:20.252 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:421: TEST_scrub_permit_time: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-22T21:12:20.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-22T21:12:20.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T21:12:20.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T21:12:20.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T21:12:20.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-22T21:12:20.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-22T21:12:20.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T21:12:20.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:12:20.271 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:12:20.271 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:12:20.271 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:12:20.271 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:12:20.272 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:12:20.273 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-22T21:12:20.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T21:12:20.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T21:12:20.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:12:20.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:12:20.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T21:12:20.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T21:12:20.300 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:12:20.300 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:12:20.300 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:12:20.300 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:12:20.300 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:12:20.300 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:12:20.301 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T21:12:20.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:12:20.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get fsid 2026-02-22T21:12:20.357 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T21:12:20.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:12:20.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:12:20.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T21:12:20.358 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T21:12:20.359 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:12:20.359 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:12:20.359 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:12:20.359 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:12:20.359 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:12:20.359 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:12:20.359 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T21:12:20.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:12:20.360 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get mon_host 2026-02-22T21:12:20.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:422: TEST_scrub_permit_time: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-22T21:12:20.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-22T21:12:20.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T21:12:20.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T21:12:20.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T21:12:20.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-22T21:12:20.418 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:12:20.514 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T21:12:20.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:12:20.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:12:20.515 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:12:20.515 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:12:20.515 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:12:20.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:12:20.516 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:12:20.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-22T21:12:20.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:423: TEST_scrub_permit_time: date -d '2 hour ago' +%H 2026-02-22T21:12:20.535 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:423: TEST_scrub_permit_time: sed 's/^0//' 2026-02-22T21:12:20.537 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:423: TEST_scrub_permit_time: local scrub_begin_hour=19 2026-02-22T21:12:20.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:424: TEST_scrub_permit_time: date -d '1 hour ago' +%H 2026-02-22T21:12:20.538 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:424: TEST_scrub_permit_time: sed 's/^0//' 2026-02-22T21:12:20.540 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:424: TEST_scrub_permit_time: local scrub_end_hour=20 2026-02-22T21:12:20.540 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: expr 3 - 1 2026-02-22T21:12:20.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: seq 0 2 2026-02-22T21:12:20.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:12:20.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:427: TEST_scrub_permit_time: run_osd td/osd-scrub-test 0 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=19 --osd_scrub_end_hour=20 2026-02-22T21:12:20.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:12:20.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:12:20.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T21:12:20.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:12:20.542 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:12:20.543 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=19 --osd_scrub_end_hour=20' 2026-02-22T21:12:20.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-22T21:12:20.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:12:20.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=27237eb0-3f48-498a-bf84-624e559ea038 2026-02-22T21:12:20.546 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 27237eb0-3f48-498a-bf84-624e559ea038' 2026-02-22T21:12:20.546 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 27237eb0-3f48-498a-bf84-624e559ea038 2026-02-22T21:12:20.547 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:12:20.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA0cZtpZ57eIBAATA4JDlklCNhSO8O32qoptA== 2026-02-22T21:12:20.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA0cZtpZ57eIBAATA4JDlklCNhSO8O32qoptA=="}' 2026-02-22T21:12:20.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 27237eb0-3f48-498a-bf84-624e559ea038 -i td/osd-scrub-test/0/new.json 2026-02-22T21:12:20.645 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:12:20.654 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-22T21:12:20.655 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=19 --osd_scrub_end_hour=20 --mkfs --key AQA0cZtpZ57eIBAATA4JDlklCNhSO8O32qoptA== --osd-uuid 27237eb0-3f48-498a-bf84-624e559ea038 2026-02-22T21:12:20.672 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:20.663+0000 7fab87abea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:20.673 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:20.667+0000 7fab87abea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:20.674 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:20.667+0000 7fab87abea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:20.674 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:20.667+0000 7fab87abea40 -1 bdev(0x55d29080ec00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:12:20.674 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:20.667+0000 7fab87abea40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-22T21:12:21.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-22T21:12:21.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:12:21.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:12:21.444 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T21:12:21.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:12:21.636 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T21:12:21.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T21:12:21.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=19 --osd_scrub_end_hour=20 2026-02-22T21:12:21.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:12:21.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:12:21.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:12:21.662 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:21.655+0000 7faf6f1b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:21.664 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:21.655+0000 7faf6f1b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:21.665 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:21.659+0000 7faf6f1b7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:21.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T21:12:21.800 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:12:21.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T21:12:21.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:12:21.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:12:21.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:12:21.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:12:21.801 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:12:21.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:12:21.801 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:12:21.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:12:22.042 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:22.035+0000 7faf6f1b7a40 -1 Falling back to public interface 2026-02-22T21:12:22.630 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:22.623+0000 7faf6f1b7a40 -1 osd.0 0 log_to_monitors true 2026-02-22T21:12:22.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:12:22.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:12:22.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:12:22.967 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:12:22.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:12:22.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:12:23.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:12:24.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:12:24.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:12:24.125 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:12:24.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:12:24.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:12:24.126 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:12:24.278 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3446449711,v1:127.0.0.1:6803/3446449711] [v2:127.0.0.1:6804/3446449711,v1:127.0.0.1:6805/3446449711] exists,up 27237eb0-3f48-498a-bf84-624e559ea038 2026-02-22T21:12:24.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:12:24.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:12:24.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:12:24.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:12:24.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:427: TEST_scrub_permit_time: run_osd td/osd-scrub-test 1 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=19 --osd_scrub_end_hour=20 2026-02-22T21:12:24.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:12:24.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:12:24.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T21:12:24.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:12:24.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:12:24.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:12:24.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:12:24.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:12:24.282 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:12:24.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:12:24.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:12:24.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:12:24.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:12:24.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:12:24.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:12:24.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:12:24.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:12:24.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=19 --osd_scrub_end_hour=20' 2026-02-22T21:12:24.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-22T21:12:24.283 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:12:24.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ac68aa2a-36e4-4e03-9594-7570ca84d97b 2026-02-22T21:12:24.284 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ac68aa2a-36e4-4e03-9594-7570ca84d97b' 2026-02-22T21:12:24.284 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 ac68aa2a-36e4-4e03-9594-7570ca84d97b 2026-02-22T21:12:24.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:12:24.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA4cZtpXqxIERAAdBHOJd0j5lzIWXKtCWwpuQ== 2026-02-22T21:12:24.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA4cZtpXqxIERAAdBHOJd0j5lzIWXKtCWwpuQ=="}' 2026-02-22T21:12:24.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ac68aa2a-36e4-4e03-9594-7570ca84d97b -i td/osd-scrub-test/1/new.json 2026-02-22T21:12:24.455 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:12:24.464 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-22T21:12:24.465 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=19 --osd_scrub_end_hour=20 --mkfs --key AQA4cZtpXqxIERAAdBHOJd0j5lzIWXKtCWwpuQ== --osd-uuid ac68aa2a-36e4-4e03-9594-7570ca84d97b 2026-02-22T21:12:24.482 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:24.475+0000 7f9e8a067a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:24.484 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:24.475+0000 7f9e8a067a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:24.485 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:24.479+0000 7f9e8a067a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:24.485 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:24.479+0000 7f9e8a067a40 -1 bdev(0x557d413a6c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:12:24.485 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:24.479+0000 7f9e8a067a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-22T21:12:25.146 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-22T21:12:25.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:12:25.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:12:25.148 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T21:12:25.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:12:25.347 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T21:12:25.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T21:12:25.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=19 --osd_scrub_end_hour=20 2026-02-22T21:12:25.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:12:25.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:12:25.348 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:12:25.371 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:25.363+0000 7fcb77b09a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:25.373 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:25.367+0000 7fcb77b09a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:25.374 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:25.367+0000 7fcb77b09a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:25.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T21:12:25.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:12:25.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T21:12:25.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:12:25.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:12:25.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:12:25.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:12:25.507 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:12:25.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:12:25.507 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:12:25.665 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:12:25.734 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:25.727+0000 7fcb77b09a40 -1 Falling back to public interface 2026-02-22T21:12:26.287 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:26.279+0000 7fcb77b09a40 -1 osd.1 0 log_to_monitors true 2026-02-22T21:12:26.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:12:26.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:12:26.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:12:26.668 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:12:26.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:12:26.668 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:12:26.824 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:12:27.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:12:27.825 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:12:27.825 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:12:27.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:12:27.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:12:27.826 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:12:27.979 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3596082607,v1:127.0.0.1:6811/3596082607] [v2:127.0.0.1:6812/3596082607,v1:127.0.0.1:6813/3596082607] exists,up ac68aa2a-36e4-4e03-9594-7570ca84d97b 2026-02-22T21:12:27.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:12:27.984 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:12:27.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:12:27.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:425: TEST_scrub_permit_time: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:12:27.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:427: TEST_scrub_permit_time: run_osd td/osd-scrub-test 2 --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=19 --osd_scrub_end_hour=20 2026-02-22T21:12:27.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:12:27.985 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:12:27.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T21:12:27.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:12:27.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-22T21:12:27.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:12:27.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:12:27.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:12:27.986 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:12:27.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-22T21:12:27.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-22T21:12:27.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:12:27.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:12:27.987 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:12:27.987 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:12:27.988 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:12:27.988 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:12:27.988 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:12:27.988 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:12:27.988 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:12:27.988 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:12:27.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:12:27.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:12:27.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:12:27.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:12:27.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:12:27.989 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:12:27.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:12:27.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:12:27.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:12:27.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:12:27.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:12:27.990 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=19 --osd_scrub_end_hour=20' 2026-02-22T21:12:27.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-22T21:12:27.991 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:12:27.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d822651c-4539-41ac-ba0f-7d07b38d2afe 2026-02-22T21:12:27.991 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 d822651c-4539-41ac-ba0f-7d07b38d2afe' 2026-02-22T21:12:27.992 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 d822651c-4539-41ac-ba0f-7d07b38d2afe 2026-02-22T21:12:27.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:12:27.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA7cZtpj7D6OhAAd54UdP62NfhaED564GTW7w== 2026-02-22T21:12:27.995 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA7cZtpj7D6OhAAd54UdP62NfhaED564GTW7w=="}' 2026-02-22T21:12:27.996 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d822651c-4539-41ac-ba0f-7d07b38d2afe -i td/osd-scrub-test/2/new.json 2026-02-22T21:12:28.151 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:12:28.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-22T21:12:28.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=19 --osd_scrub_end_hour=20 --mkfs --key AQA7cZtpj7D6OhAAd54UdP62NfhaED564GTW7w== --osd-uuid d822651c-4539-41ac-ba0f-7d07b38d2afe 2026-02-22T21:12:28.178 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:28.171+0000 7f738d762a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:28.180 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:28.171+0000 7f738d762a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:28.181 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:28.175+0000 7f738d762a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:28.181 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:28.175+0000 7f738d762a40 -1 bdev(0x55afa99bec00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:12:28.181 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:28.175+0000 7f738d762a40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-22T21:12:28.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-22T21:12:28.830 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:12:28.831 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T21:12:28.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:12:28.831 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:12:29.031 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T21:12:29.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T21:12:29.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --bluestore_cache_autotune=false --osd_deep_scrub_randomize_ratio=0.0 --osd_scrub_interval_randomize_ratio=0 --osd_scrub_begin_hour=19 --osd_scrub_end_hour=20 2026-02-22T21:12:29.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:12:29.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:12:29.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:12:29.057 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:29.051+0000 7f7ce21f2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:29.059 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:29.051+0000 7f7ce21f2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:29.060 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:29.051+0000 7f7ce21f2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:12:29.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T21:12:29.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:12:29.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T21:12:29.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:12:29.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:12:29.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:12:29.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:12:29.190 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:12:29.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:12:29.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:12:29.349 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:12:29.490 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:29.483+0000 7f7ce21f2a40 -1 Falling back to public interface 2026-02-22T21:12:30.118 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:12:30.111+0000 7f7ce21f2a40 -1 osd.2 0 log_to_monitors true 2026-02-22T21:12:30.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:12:30.350 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:12:30.351 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:12:30.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:12:30.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:12:30.351 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:12:30.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:12:31.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:12:31.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:12:31.513 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:12:31.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:12:31.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:12:31.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:12:31.668 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1620624645,v1:127.0.0.1:6819/1620624645] [v2:127.0.0.1:6820/1620624645,v1:127.0.0.1:6821/1620624645] exists,up d822651c-4539-41ac-ba0f-7d07b38d2afe 2026-02-22T21:12:31.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:12:31.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:12:31.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:12:31.669 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:435: TEST_scrub_permit_time: create_pool test 1 1 2026-02-22T21:12:31.670 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-22T21:12:31.870 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T21:12:31.880 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T21:12:32.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:436: TEST_scrub_permit_time: wait_for_clean 2026-02-22T21:12:32.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:12:32.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:12:32.882 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:12:32.883 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:12:32.883 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:12:32.883 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:12:32.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:12:32.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:12:32.884 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:12:32.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:12:32.997 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:12:32.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:12:32.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:12:32.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:12:32.998 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:12:33.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:12:33.157 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:12:33.157 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:12:33.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:12:33.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:12:33.157 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:12:33.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-22T21:12:33.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-22T21:12:33.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T21:12:33.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:12:33.228 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:12:33.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-22T21:12:33.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-22T21:12:33.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672967' 2026-02-22T21:12:33.299 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:12:33.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:12:33.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509444 2026-02-22T21:12:33.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509444 2026-02-22T21:12:33.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672967 2-64424509444' 2026-02-22T21:12:33.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:12:33.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-22T21:12:33.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:12:33.375 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:12:33.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-22T21:12:33.376 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:12:33.378 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-22T21:12:33.378 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836491 2026-02-22T21:12:33.379 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T21:12:33.379 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:12:33.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T21:12:33.543 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:12:34.544 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:12:34.545 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:12:34.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T21:12:34.708 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:12:35.709 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-22T21:12:35.709 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:12:35.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836491 2026-02-22T21:12:35.866 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:12:35.867 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-22T21:12:35.867 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:12:35.868 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:12:35.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-22T21:12:35.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:12:35.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-22T21:12:35.871 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-22T21:12:35.871 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672967 2026-02-22T21:12:35.871 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:12:36.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672969 -lt 42949672967 2026-02-22T21:12:36.038 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:12:36.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509444 2026-02-22T21:12:36.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:12:36.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:12:36.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509444 2026-02-22T21:12:36.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:12:36.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509444 2026-02-22T21:12:36.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2026-02-22T21:12:36.043 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509444 2026-02-22T21:12:36.043 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:12:36.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509445 -lt 64424509444 2026-02-22T21:12:36.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:12:36.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:12:36.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:12:36.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:12:36.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:12:36.403 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:12:36.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:12:36.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:12:36.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:12:36.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:12:36.404 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:12:36.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:12:36.560 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:12:36.560 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:12:36.561 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:12:36.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:12:36.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:12:36.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:12:36.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:439: TEST_scrub_permit_time: get_pg test SOMETHING 2026-02-22T21:12:36.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2026-02-22T21:12:36.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=SOMETHING 2026-02-22T21:12:36.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test SOMETHING 2026-02-22T21:12:36.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-22T21:12:36.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:439: TEST_scrub_permit_time: local pgid=1.0 2026-02-22T21:12:36.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: get_primary test SOMETHING 2026-02-22T21:12:36.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-22T21:12:36.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=SOMETHING 2026-02-22T21:12:36.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test SOMETHING 2026-02-22T21:12:36.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T21:12:37.071 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:440: TEST_scrub_permit_time: local primary=1 2026-02-22T21:12:37.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:441: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:37.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:37.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:37.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:37.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:37.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:441: TEST_scrub_permit_time: local last_scrub=2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:37.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:446: TEST_scrub_permit_time: ceph tell 1.0 schedule-scrub 86400 2026-02-22T21:12:37.290 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:12:37.290 INFO:tasks.workunit.client.0.trial150.stdout: "deep": false, 2026-02-22T21:12:37.290 INFO:tasks.workunit.client.0.trial150.stdout: "must": false, 2026-02-22T21:12:37.290 INFO:tasks.workunit.client.0.trial150.stdout: "stamp": "2026-02-21T21:12:37.285501+0000" 2026-02-22T21:12:37.291 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:12:37.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i=0 )) 2026-02-22T21:12:37.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:37.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:37.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:37.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:37.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:37.299 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:37.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-22T21:12:31.864616+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:37.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:38.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:38.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:38.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:38.460 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:38.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:38.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:38.461 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:38.619 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-22T21:12:31.864616+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:38.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:39.620 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:39.621 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:39.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:39.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:39.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:39.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:39.622 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:39.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:39.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:40.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:40.779 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:40.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:40.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:40.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:40.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:40.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:40.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:40.936 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:41.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:41.937 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:41.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:41.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:41.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:41.938 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:41.939 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:42.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:42.095 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:43.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:43.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:43.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:43.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:43.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:43.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:43.097 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:43.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:43.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:44.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:44.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:44.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:44.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:44.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:44.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:44.256 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:44.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:44.415 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:45.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:45.417 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:45.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:45.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:45.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:45.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:45.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:45.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:45.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:46.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:46.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:46.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:46.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:46.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:46.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:46.580 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:46.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:46.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:47.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:47.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:47.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:47.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:47.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:47.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:47.738 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:47.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:47.892 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:48.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:48.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:48.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:48.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:48.894 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:48.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:48.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:49.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:49.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:50.050 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:50.051 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:50.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:50.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:50.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:50.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:50.051 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:50.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:50.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:51.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:51.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:51.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:51.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:51.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:51.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:51.210 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:51.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:51.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:52.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:52.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:52.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:52.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:52.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:52.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:52.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:52.528 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:52.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:53.529 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:53.530 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:53.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:53.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:53.530 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:53.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:53.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:53.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:53.690 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:54.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:54.691 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:54.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:54.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:54.691 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:54.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:54.692 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:54.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:54.848 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:55.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:55.849 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:55.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:55.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:55.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:55.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:55.850 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:56.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:56.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:57.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:57.008 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:57.009 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:57.009 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:57.009 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:57.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:57.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:57.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:57.163 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:58.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:58.164 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:58.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:58.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:58.165 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:58.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:58.166 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:58.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:58.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:12:59.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:12:59.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:12:59.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:12:59.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:12:59.325 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:12:59.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:12:59.326 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:12:59.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:12:59.483 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:13:00.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:13:00.484 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:13:00.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:13:00.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:00.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:00.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:00.486 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:00.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:13:00.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:13:01.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:13:01.645 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:13:01.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:13:01.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:01.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:01.646 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:01.647 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:01.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:13:01.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:13:02.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:13:02.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:13:02.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:13:02.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:02.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:02.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:02.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:02.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:13:02.964 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:13:03.965 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:13:03.966 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:13:03.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:13:03.966 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:03.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:03.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:03.967 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:04.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:13:04.124 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:13:05.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:13:05.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:13:05.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:13:05.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:05.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:05.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:05.126 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:05.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:13:05.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:13:06.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:13:06.283 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:13:06.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:13:06.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:06.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:06.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:06.284 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:06.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:13:06.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:13:07.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:13:07.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:13:07.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:13:07.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:07.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:07.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:07.440 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:07.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:13:07.595 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:13:08.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:13:08.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:13:08.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:13:08.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:08.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:08.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:08.598 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:13:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:13:09.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:13:09.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:13:09.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:13:09.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:09.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:09.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:09.755 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:09.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:13:09.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:13:10.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:13:10.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:13:10.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: get_last_scrub_stamp 1.0 2026-02-22T21:13:10.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:10.912 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:10.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:10.913 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:11.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:450: TEST_scrub_permit_time: test 2026-02-21T21:12:37.285501+0000 '>' 2026-02-22T21:12:31.864616+0000 2026-02-22T21:13:11.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:453: TEST_scrub_permit_time: sleep 1 2026-02-22T21:13:12.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i++ )) 2026-02-22T21:13:12.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:449: TEST_scrub_permit_time: (( i < 30 )) 2026-02-22T21:13:12.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:455: TEST_scrub_permit_time: perf_counters td/osd-scrub-test 3 2026-02-22T21:13:12.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-22T21:13:12.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-22T21:13:12.070 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-22T21:13:12.072 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-22T21:13:12.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:13:12.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-22T21:13:12.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:13:12.151 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:13:12.151 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:13:12.151 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:13:12.151 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:13:12.151 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:13:12.151 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:13:12.151 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:13:12.151 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:13:12.151 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:13:12.152 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:13:12.153 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:13:12.154 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:13:12.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:13:12.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-22T21:13:12.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:13:12.223 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:13:12.223 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:13:12.223 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:13:12.223 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:13:12.223 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:13:12.223 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:13:12.223 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:13:12.223 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:13:12.223 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:13:12.224 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:13:12.224 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:13:12.224 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:13:12.224 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:13:12.224 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.224 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.224 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:13:12.224 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:13:12.224 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:13:12.224 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:13:12.224 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:13:12.224 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:13:12.225 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:13:12.226 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:13:12.226 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:13:12.226 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:13:12.226 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:13:12.226 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:13:12.226 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:13:12.226 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:13:12.226 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:13:12.226 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:13:12.226 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.226 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.226 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:13:12.227 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.228 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.228 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:13:12.228 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:13:12.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:13:12.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-22T21:13:12.229 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:13:12.294 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:13:12.295 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:13:12.295 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:13:12.295 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:13:12.295 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:13:12.295 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:13:12.295 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:13:12.295 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:13:12.295 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:13:12.295 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:13:12.295 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:13:12.295 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:13:12.296 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:13:12.296 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.296 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.296 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:13:12.296 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:13:12.296 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:13:12.296 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:13:12.296 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:13:12.296 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:13:12.296 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:13:12.296 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:13:12.296 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:13:12.297 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:13:12.297 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:13:12.297 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:13:12.297 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:13:12.297 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.297 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.297 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:13:12.297 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:13:12.297 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:13:12.297 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:13:12.297 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:13:12.297 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:13:12.298 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:13:12.299 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:13:12.299 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:13:12.299 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:13:12.299 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:13:12.299 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:13:12.299 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:13:12.299 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:13:12.299 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:13:12.299 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:13:12.299 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.299 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:13:12.299 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:13:12.300 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:13:12.300 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Teardown Test TEST_scrub_permit_time ------------------ 2026-02-22T21:13:12.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_scrub_permit_time ------------------' 2026-02-22T21:13:12.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-22T21:13:12.301 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:13:12.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:13:12.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:13:12.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:13:12.302 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:13:12.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:13:12.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:13:12.302 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:13:12.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:13:12.406 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:13:12.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:13:12.407 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:13:12.408 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:13:12.409 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:13:12.409 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:13:12.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:13:12.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:13:12.410 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:13:12.410 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:13:12.411 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:13:12.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:13:12.413 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:13:12.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:13:12.424 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:13:12.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:13:12.425 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:13:12.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:13:12.426 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:13:12.426 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Complete Test TEST_scrub_permit_time ------------------ 2026-02-22T21:13:12.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_scrub_permit_time ------------------' 2026-02-22T21:13:12.427 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:31: run: for func in $funcs 2026-02-22T21:13:12.427 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Prepare Test TEST_scrub_test ------------------- 2026-02-22T21:13:12.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:32: run: echo '-------------- Prepare Test TEST_scrub_test -------------------' 2026-02-22T21:13:12.428 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:33: run: setup td/osd-scrub-test 2026-02-22T21:13:12.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-scrub-test 2026-02-22T21:13:12.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-scrub-test 2026-02-22T21:13:12.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:13:12.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:13:12.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:13:12.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:13:12.429 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:13:12.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:13:12.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:13:12.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:13:12.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:13:12.430 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:13:12.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:13:12.431 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:13:12.432 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:13:12.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:13:12.433 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:13:12.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:13:12.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:13:12.435 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:13:12.435 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:13:12.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:13:12.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:13:12.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:13:12.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:13:12.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:13:12.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:13:12.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:13:12.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:13:12.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:13:12.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-scrub-test 2026-02-22T21:13:12.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T21:13:12.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:13:12.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:13:12.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.516403 2026-02-22T21:13:12.444 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T21:13:12.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-22T21:13:12.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T21:13:12.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-scrub-test 1' TERM HUP INT 2026-02-22T21:13:12.445 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Run Test TEST_scrub_test ----------------------- 2026-02-22T21:13:12.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:34: run: echo '-------------- Run Test TEST_scrub_test -----------------------' 2026-02-22T21:13:12.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:35: run: TEST_scrub_test td/osd-scrub-test 2026-02-22T21:13:12.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:52: TEST_scrub_test: local dir=td/osd-scrub-test 2026-02-22T21:13:12.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:53: TEST_scrub_test: local poolname=test 2026-02-22T21:13:12.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:54: TEST_scrub_test: local OSDS=3 2026-02-22T21:13:12.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:55: TEST_scrub_test: local objects=15 2026-02-22T21:13:12.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:57: TEST_scrub_test: run_mon td/osd-scrub-test a --osd_pool_default_size=3 2026-02-22T21:13:12.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-scrub-test 2026-02-22T21:13:12.447 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T21:13:12.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T21:13:12.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T21:13:12.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-scrub-test/a 2026-02-22T21:13:12.448 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-scrub-test/a --run-dir=td/osd-scrub-test --osd_pool_default_size=3 2026-02-22T21:13:12.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T21:13:12.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:13:12.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:13:12.469 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:13:12.469 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:13:12.469 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:13:12.469 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:13:12.470 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-scrub-test/a '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --mon-cluster-log-file=td/osd-scrub-test/log --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2026-02-22T21:13:12.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T21:13:12.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T21:13:12.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:13:12.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:13:12.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T21:13:12.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T21:13:12.498 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:13:12.498 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:13:12.498 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:13:12.498 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:13:12.498 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:13:12.498 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:13:12.498 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T21:13:12.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:13:12.498 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get fsid 2026-02-22T21:13:12.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T21:13:12.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:13:12.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:13:12.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T21:13:12.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T21:13:12.556 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:13:12.556 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:13:12.556 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:13:12.556 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:13:12.556 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:13:12.556 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:13:12.557 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.516403/ceph-mon.a.asok 2026-02-22T21:13:12.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:13:12.557 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.516403/ceph-mon.a.asok config get mon_host 2026-02-22T21:13:12.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:58: TEST_scrub_test: run_mgr td/osd-scrub-test x --mgr_stats_period=1 2026-02-22T21:13:12.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-scrub-test 2026-02-22T21:13:12.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T21:13:12.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T21:13:12.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T21:13:12.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-scrub-test/x 2026-02-22T21:13:12.611 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:13:12.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T21:13:12.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:13:12.702 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:13:12.702 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:13:12.702 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:13:12.702 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:13:12.703 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:13:12.703 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:13:12.705 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-scrub-test/x '--log-file=td/osd-scrub-test/$name.log' '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --run-dir=td/osd-scrub-test '--pid-file=td/osd-scrub-test/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-22T21:13:12.722 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:59: TEST_scrub_test: local 'ceph_osd_args=--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 ' 2026-02-22T21:13:12.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:60: TEST_scrub_test: ceph_osd_args+='--osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 ' 2026-02-22T21:13:12.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:61: TEST_scrub_test: ceph_osd_args+=--osd_stats_update_period_scrubbing=2 2026-02-22T21:13:12.723 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: expr 3 - 1 2026-02-22T21:13:12.724 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: seq 0 2 2026-02-22T21:13:12.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:13:12.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: run_osd td/osd-scrub-test 0 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/0 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/0' 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/0/journal' 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:13:12.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:13:12.728 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:13:12.728 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:13:12.728 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:13:12.729 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:13:12.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:13:12.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:13:12.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:13:12.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:13:12.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:13:12.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:13:12.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:13:12.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:13:12.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:13:12.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:13:12.729 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:13:12.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2026-02-22T21:13:12.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/0 2026-02-22T21:13:12.730 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:13:12.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d4bfc324-10c8-4581-8ac0-09138478e4d2 2026-02-22T21:13:12.730 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d4bfc324-10c8-4581-8ac0-09138478e4d2' 2026-02-22T21:13:12.730 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 d4bfc324-10c8-4581-8ac0-09138478e4d2 2026-02-22T21:13:12.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:13:12.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBocZtpOtnDKxAA89M2Gfvy4gT80VswtygW7g== 2026-02-22T21:13:12.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBocZtpOtnDKxAA89M2Gfvy4gT80VswtygW7g=="}' 2026-02-22T21:13:12.740 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d4bfc324-10c8-4581-8ac0-09138478e4d2 -i td/osd-scrub-test/0/new.json 2026-02-22T21:13:12.826 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:13:12.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/0/new.json 2026-02-22T21:13:12.835 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBocZtpOtnDKxAA89M2Gfvy4gT80VswtygW7g== --osd-uuid d4bfc324-10c8-4581-8ac0-09138478e4d2 2026-02-22T21:13:12.852 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:12.843+0000 7f00f48a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:12.854 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:12.847+0000 7f00f48a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:12.854 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:12.847+0000 7f00f48a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:12.855 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:12.847+0000 7f00f48a1a40 -1 bdev(0x5584c1bdec00 td/osd-scrub-test/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:13:12.855 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:12.847+0000 7f00f48a1a40 -1 bluestore(td/osd-scrub-test/0) _read_fsid unparsable uuid 2026-02-22T21:13:13.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/0/keyring 2026-02-22T21:13:13.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:13:13.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:13:13.638 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T21:13:13.638 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:13:13.841 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T21:13:13.841 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T21:13:13.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/0 --osd-journal=td/osd-scrub-test/0/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-22T21:13:13.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:13:13.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:13:13.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:13:13.866 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:13.859+0000 7f813dadfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:13.868 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:13.859+0000 7f813dadfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:13.869 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:13.863+0000 7f813dadfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:13.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T21:13:13.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:13:13.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T21:13:13.998 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:13:13.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:13:13.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:13:13.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:13:13.999 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:13:14.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:13:14.000 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:13:14.157 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:13:14.226 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:14.219+0000 7f813dadfa40 -1 Falling back to public interface 2026-02-22T21:13:14.774 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:14.767+0000 7f813dadfa40 -1 osd.0 0 log_to_monitors true 2026-02-22T21:13:15.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:13:15.159 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:13:15.159 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:13:15.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:13:15.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:13:15.160 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:13:15.322 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:13:16.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:13:16.324 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:13:16.325 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:13:16.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:13:16.325 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:13:16.326 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:13:16.489 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3872614068,v1:127.0.0.1:6803/3872614068] [v2:127.0.0.1:6804/3872614068,v1:127.0.0.1:6805/3872614068] exists,up d4bfc324-10c8-4581-8ac0-09138478e4d2 2026-02-22T21:13:16.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:13:16.492 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: run_osd td/osd-scrub-test 1 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/1 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:13:16.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/1' 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/1/journal' 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:13:16.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:13:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:13:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:13:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:13:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:13:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:13:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:13:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:13:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:13:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:13:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2026-02-22T21:13:16.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/1 2026-02-22T21:13:16.496 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:13:16.496 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dc01e3d6-8473-4523-acd2-63e59d9a0a8c 2026-02-22T21:13:16.496 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 dc01e3d6-8473-4523-acd2-63e59d9a0a8c 2026-02-22T21:13:16.497 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 dc01e3d6-8473-4523-acd2-63e59d9a0a8c' 2026-02-22T21:13:16.497 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:13:16.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBscZtpaqbZHRAAQxrIZDyaSBBjyQe3A214mw== 2026-02-22T21:13:16.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBscZtpaqbZHRAAQxrIZDyaSBBjyQe3A214mw=="}' 2026-02-22T21:13:16.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dc01e3d6-8473-4523-acd2-63e59d9a0a8c -i td/osd-scrub-test/1/new.json 2026-02-22T21:13:16.669 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:13:16.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/1/new.json 2026-02-22T21:13:16.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBscZtpaqbZHRAAQxrIZDyaSBBjyQe3A214mw== --osd-uuid dc01e3d6-8473-4523-acd2-63e59d9a0a8c 2026-02-22T21:13:16.696 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:16.687+0000 7fb5c2fd4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:16.698 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:16.691+0000 7fb5c2fd4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:16.699 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:16.691+0000 7fb5c2fd4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:16.699 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:16.691+0000 7fb5c2fd4a40 -1 bdev(0x55b8b5bd0c00 td/osd-scrub-test/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:13:16.699 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:16.691+0000 7fb5c2fd4a40 -1 bluestore(td/osd-scrub-test/1) _read_fsid unparsable uuid 2026-02-22T21:13:17.367 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/1/keyring 2026-02-22T21:13:17.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:13:17.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:13:17.369 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T21:13:17.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:13:17.575 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T21:13:17.576 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T21:13:17.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/1 --osd-journal=td/osd-scrub-test/1/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-22T21:13:17.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:13:17.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:13:17.577 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:13:17.601 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:17.595+0000 7fa541da6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:17.603 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:17.595+0000 7fa541da6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:17.604 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:17.595+0000 7fa541da6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:17.732 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T21:13:17.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:13:17.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T21:13:17.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:13:17.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:13:17.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:13:17.733 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:13:17.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:13:17.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:13:17.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:13:17.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:13:17.990 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:17.983+0000 7fa541da6a40 -1 Falling back to public interface 2026-02-22T21:13:18.603 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:18.595+0000 7fa541da6a40 -1 osd.1 0 log_to_monitors true 2026-02-22T21:13:18.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:13:18.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:13:18.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:13:18.897 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:13:18.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:13:18.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:13:19.058 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:13:20.059 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:13:20.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:13:20.060 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:13:20.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:13:20.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:13:20.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:13:20.218 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2573306977,v1:127.0.0.1:6811/2573306977] [v2:127.0.0.1:6812/2573306977,v1:127.0.0.1:6813/2573306977] exists,up dc01e3d6-8473-4523-acd2-63e59d9a0a8c 2026-02-22T21:13:20.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:13:20.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:13:20.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:13:20.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:62: TEST_scrub_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:13:20.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:64: TEST_scrub_test: run_osd td/osd-scrub-test 2 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-22T21:13:20.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-scrub-test 2026-02-22T21:13:20.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:13:20.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T21:13:20.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-scrub-test/2 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:13:20.220 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:13:20.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:13:20.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:13:20.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:13:20.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:13:20.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:13:20.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:13:20.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:13:20.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:13:20.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:13:20.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:13:20.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:13:20.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:13:20.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2026-02-22T21:13:20.222 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-scrub-test/2 2026-02-22T21:13:20.222 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:13:20.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d1e39661-9856-4012-a977-1fff22eacce6 2026-02-22T21:13:20.223 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 d1e39661-9856-4012-a977-1fff22eacce6 2026-02-22T21:13:20.223 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 d1e39661-9856-4012-a977-1fff22eacce6' 2026-02-22T21:13:20.223 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:13:20.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBwcZtpwV2wDRAAhqTpFUoIft12i+ngyMq98g== 2026-02-22T21:13:20.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBwcZtpwV2wDRAAhqTpFUoIft12i+ngyMq98g=="}' 2026-02-22T21:13:20.235 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d1e39661-9856-4012-a977-1fff22eacce6 -i td/osd-scrub-test/2/new.json 2026-02-22T21:13:20.395 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:13:20.406 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-scrub-test/2/new.json 2026-02-22T21:13:20.407 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 --mkfs --key AQBwcZtpwV2wDRAAhqTpFUoIft12i+ngyMq98g== --osd-uuid d1e39661-9856-4012-a977-1fff22eacce6 2026-02-22T21:13:20.431 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:20.423+0000 7f1b2d45aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:20.433 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:20.427+0000 7f1b2d45aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:20.434 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:20.427+0000 7f1b2d45aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:20.434 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:20.427+0000 7f1b2d45aa40 -1 bdev(0x55d9a5880c00 td/osd-scrub-test/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:13:20.434 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:20.427+0000 7f1b2d45aa40 -1 bluestore(td/osd-scrub-test/2) _read_fsid unparsable uuid 2026-02-22T21:13:21.043 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-scrub-test/2/keyring 2026-02-22T21:13:21.044 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:13:21.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:13:21.045 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T21:13:21.045 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-scrub-test/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:13:21.245 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T21:13:21.246 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T21:13:21.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-22T21:13:21.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:13:21.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:13:21.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:13:21.271 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:21.263+0000 7f86b9d19a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:21.273 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:21.267+0000 7f86b9d19a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:21.274 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:21.267+0000 7f86b9d19a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:21.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T21:13:21.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:13:21.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T21:13:21.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:13:21.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:13:21.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:13:21.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:13:21.405 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:13:21.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:13:21.405 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:13:21.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:13:21.650 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:21.643+0000 7f86b9d19a40 -1 Falling back to public interface 2026-02-22T21:13:22.287 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:22.279+0000 7f86b9d19a40 -1 osd.2 0 log_to_monitors true 2026-02-22T21:13:22.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:13:22.566 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:13:22.566 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:13:22.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:13:22.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:13:22.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:13:22.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:13:23.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:13:23.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:13:23.720 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:13:23.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:13:23.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:13:23.720 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:13:23.877 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1744715304,v1:127.0.0.1:6819/1744715304] [v2:127.0.0.1:6820/1744715304,v1:127.0.0.1:6821/1744715304] exists,up d1e39661-9856-4012-a977-1fff22eacce6 2026-02-22T21:13:23.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:13:23.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:13:23.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:13:23.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:68: TEST_scrub_test: create_pool test 1 1 2026-02-22T21:13:23.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2026-02-22T21:13:24.086 INFO:tasks.workunit.client.0.trial150.stderr:pool 'test' created 2026-02-22T21:13:24.097 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-22T21:13:25.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:69: TEST_scrub_test: wait_for_clean 2026-02-22T21:13:25.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:13:25.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:13:25.099 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:13:25.099 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:13:25.100 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:13:25.100 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:13:25.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:13:25.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:13:25.101 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:13:25.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:13:25.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:13:25.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:13:25.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:13:25.211 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:13:25.212 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:13:25.373 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:13:25.374 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:13:25.374 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:13:25.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:13:25.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:25.374 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:13:25.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-22T21:13:25.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-22T21:13:25.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491' 2026-02-22T21:13:25.444 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:25.445 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:13:25.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-22T21:13:25.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-22T21:13:25.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672967' 2026-02-22T21:13:25.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:25.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:13:25.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509444 2026-02-22T21:13:25.592 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509444 2026-02-22T21:13:25.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836491 1-42949672967 2-64424509444' 2026-02-22T21:13:25.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:25.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-22T21:13:25.593 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:25.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:13:25.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836491 2026-02-22T21:13:25.595 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:25.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836491 2026-02-22T21:13:25.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-22T21:13:25.596 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836491 2026-02-22T21:13:25.597 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:25.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T21:13:25.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:13:26.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:13:26.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:26.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836490 -lt 21474836491 2026-02-22T21:13:26.919 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:13:27.921 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-22T21:13:27.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:28.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836493 -lt 21474836491 2026-02-22T21:13:28.081 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:28.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-22T21:13:28.082 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:28.083 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:13:28.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672967 2026-02-22T21:13:28.084 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:28.085 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672967 2026-02-22T21:13:28.086 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672967 2026-02-22T21:13:28.086 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-22T21:13:28.086 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:13:28.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672969 -lt 42949672967 2026-02-22T21:13:28.254 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:28.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509444 2026-02-22T21:13:28.255 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:28.256 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:13:28.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509444 2026-02-22T21:13:28.257 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:28.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509444 2026-02-22T21:13:28.258 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2026-02-22T21:13:28.259 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509444 2026-02-22T21:13:28.259 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:13:28.424 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509445 -lt 64424509444 2026-02-22T21:13:28.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:13:28.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:13:28.425 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:13:28.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:13:28.613 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:13:28.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:13:28.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:13:28.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:13:28.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:13:28.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:13:28.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:13:28.769 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:13:28.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:13:28.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:13:28.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:13:28.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:13:28.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:13:28.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:13:28.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: ceph osd dump 2026-02-22T21:13:28.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: grep '^pool.*['\'']test['\'']' 2026-02-22T21:13:28.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: awk '{ print $2 }' 2026-02-22T21:13:29.112 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:70: TEST_scrub_test: poolid=1 2026-02-22T21:13:29.112 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:72: TEST_scrub_test: file_with_random_data 1032 2026-02-22T21:13:29.113 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: file_with_random_data: local size_bytes=1032 2026-02-22T21:13:29.113 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: mktemp 2026-02-22T21:13:29.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: local file=/tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: file_with_random_data: dd if=/dev/urandom of=/tmp/tmp.ki2w2WIHR4 bs=1032 count=1 2026-02-22T21:13:29.115 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T21:13:29.115 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T21:13:29.115 INFO:tasks.workunit.client.0.trial150.stderr:1032 bytes (1.0 kB, 1.0 KiB) copied, 9.8986e-05 s, 10.4 MB/s 2026-02-22T21:13:29.115 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: file_with_random_data: printf %s /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:72: TEST_scrub_test: local testdata_file=/tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: seq 1 15 2026-02-22T21:13:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.116 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj1 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.136 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj2 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj3 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj4 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.203 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj5 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj6 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.246 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj7 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj8 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj9 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.318 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj10 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.343 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj11 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.368 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj12 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.392 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj13 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.416 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj14 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:73: TEST_scrub_test: for i in `seq 1 $objects` 2026-02-22T21:13:29.433 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:75: TEST_scrub_test: rados -p test put obj15 /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.455 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:77: TEST_scrub_test: rm -f /tmp/tmp.ki2w2WIHR4 2026-02-22T21:13:29.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:79: TEST_scrub_test: get_primary test obj1 2026-02-22T21:13:29.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-22T21:13:29.457 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T21:13:29.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-22T21:13:29.458 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T21:13:29.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:79: TEST_scrub_test: local primary=1 2026-02-22T21:13:29.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:80: TEST_scrub_test: get_not_primary test obj1 2026-02-22T21:13:29.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2026-02-22T21:13:29.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2026-02-22T21:13:29.619 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2026-02-22T21:13:29.620 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2026-02-22T21:13:29.620 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2026-02-22T21:13:29.620 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2026-02-22T21:13:29.620 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T21:13:29.780 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-22T21:13:29.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2026-02-22T21:13:29.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-22T21:13:29.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:80: TEST_scrub_test: local otherosd=0 2026-02-22T21:13:29.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:81: TEST_scrub_test: '[' 0 = 2 ']' 2026-02-22T21:13:29.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:85: TEST_scrub_test: local anotherosd=2 2026-02-22T21:13:29.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:88: TEST_scrub_test: file_with_random_data 512 2026-02-22T21:13:29.940 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: file_with_random_data: local size_bytes=512 2026-02-22T21:13:29.940 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: mktemp 2026-02-22T21:13:29.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: local file=/tmp/tmp.crXaH5s3fo 2026-02-22T21:13:29.941 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: file_with_random_data: dd if=/dev/urandom of=/tmp/tmp.crXaH5s3fo bs=512 count=1 2026-02-22T21:13:29.943 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records in 2026-02-22T21:13:29.943 INFO:tasks.workunit.client.0.trial150.stderr:1+0 records out 2026-02-22T21:13:29.943 INFO:tasks.workunit.client.0.trial150.stderr:512 bytes copied, 9.7252e-05 s, 5.3 MB/s 2026-02-22T21:13:29.943 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2347: file_with_random_data: printf %s /tmp/tmp.crXaH5s3fo 2026-02-22T21:13:29.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:88: TEST_scrub_test: local corrupt_data_file=/tmp/tmp.crXaH5s3fo 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:89: TEST_scrub_test: objectstore_tool td/osd-scrub-test 2 obj1 set-bytes /tmp/tmp.crXaH5s3fo 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-scrub-test 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-scrub-test 2 obj1 set-bytes /tmp/tmp.crXaH5s3fo 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-scrub-test 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-scrub-test TERM osd.2 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:13:29.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:13:30.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:13:30.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-scrub-test 2 obj1 set-bytes /tmp/tmp.crXaH5s3fo 2026-02-22T21:13:30.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-scrub-test 2026-02-22T21:13:30.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T21:13:30.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T21:13:30.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T21:13:30.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-scrub-test/2 2026-02-22T21:13:30.253 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-scrub-test/2 obj1 set-bytes /tmp/tmp.crXaH5s3fo 2026-02-22T21:13:30.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-scrub-test 2 --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-22T21:13:30.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-scrub-test 2026-02-22T21:13:30.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T21:13:30.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T21:13:30.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T21:13:30.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-scrub-test/2 2026-02-22T21:13:30.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 ' 2026-02-22T21:13:30.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:13:30.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:13:30.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:13:30.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-scrub-test/2' 2026-02-22T21:13:30.557 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-scrub-test/2/journal' 2026-02-22T21:13:30.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:13:30.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T21:13:30.558 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-scrub-test' 2026-02-22T21:13:30.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T21:13:30.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:13:30.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:13:30.558 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:13:30.558 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:13:30.558 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:13:30.558 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:13:30.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' 2026-02-22T21:13:30.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:13:30.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-scrub-test/$name.log' 2026-02-22T21:13:30.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-scrub-test/$name.pid' 2026-02-22T21:13:30.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:13:30.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:13:30.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:13:30.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:13:30.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T21:13:30.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2' 2026-02-22T21:13:30.559 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-scrub-test/2 2026-02-22T21:13:30.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T21:13:30.560 INFO:tasks.workunit.client.0.trial150.stderr:start osd.2 2026-02-22T21:13:30.560 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=62094851-1f33-40bc-a4c4-42bb207e4df8 --auth-supported=none --mon-host=127.0.0.1:7138 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-scrub-test/2 --osd-journal=td/osd-scrub-test/2/journal --chdir= --run-dir=td/osd-scrub-test '--admin-socket=/tmp/ceph-asok.516403/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-scrub-test/$name.log' '--pid-file=td/osd-scrub-test/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-scrub-interval-randomize-ratio=0 --osd-deep-scrub-randomize-ratio=0 --osd_scrub_backoff_ratio=0 --osd_stats_update_period_not_scrubbing=3 --osd_stats_update_period_scrubbing=2 2026-02-22T21:13:30.560 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-scrub-test/2/whoami 2026-02-22T21:13:30.561 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T21:13:30.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T21:13:30.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T21:13:30.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T21:13:30.582 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:30.575+0000 7fcb7af94a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:30.584 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:30.579+0000 7fcb7af94a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:30.585 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:30.579+0000 7fcb7af94a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:13:30.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T21:13:30.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:13:30.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T21:13:30.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:13:30.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:13:30.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:13:30.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:13:30.715 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T21:13:30.715 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:13:30.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:13:30.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:13:30.950 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:30.943+0000 7fcb7af94a40 -1 Falling back to public interface 2026-02-22T21:13:31.518 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:13:31.511+0000 7fcb7af94a40 -1 osd.2 18 log_to_monitors true 2026-02-22T21:13:31.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:13:31.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 500 )) 2026-02-22T21:13:31.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:13:31.875 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:13:31.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:13:31.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:13:32.032 INFO:tasks.workunit.client.0.trial150.stderr:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 19 last_clean_interval [15,18) [v2:127.0.0.1:6818/2666220258,v1:127.0.0.1:6819/2666220258] [v2:127.0.0.1:6820/2666220258,v1:127.0.0.1:6821/2666220258] exists,up d1e39661-9856-4012-a977-1fff22eacce6 2026-02-22T21:13:32.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:13:32.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:13:32.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:13:32.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T21:13:32.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:13:32.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:13:32.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:13:32.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:13:32.034 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:13:32.034 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:13:32.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:13:32.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:13:32.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:13:32.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:13:32.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:13:32.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:13:32.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:13:32.137 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:13:32.137 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:13:32.294 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:13:32.294 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:13:32.295 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:13:32.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:13:32.295 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:32.295 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:13:32.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836499 2026-02-22T21:13:32.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836499 2026-02-22T21:13:32.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499' 2026-02-22T21:13:32.365 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:32.366 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:13:32.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672975 2026-02-22T21:13:32.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672975 2026-02-22T21:13:32.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672975' 2026-02-22T21:13:32.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:32.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:13:32.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280514 2026-02-22T21:13:32.511 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280514 2026-02-22T21:13:32.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836499 1-42949672975 2-94489280514' 2026-02-22T21:13:32.512 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:32.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836499 2026-02-22T21:13:32.512 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:32.513 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:13:32.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836499 2026-02-22T21:13:32.513 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:32.514 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836499 2026-02-22T21:13:32.515 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2026-02-22T21:13:32.515 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836499 2026-02-22T21:13:32.515 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:32.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836497 -lt 21474836499 2026-02-22T21:13:32.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:13:33.680 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:13:33.681 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:33.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836500 -lt 21474836499 2026-02-22T21:13:33.842 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:33.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672975 2026-02-22T21:13:33.843 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:33.844 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:13:33.845 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672975 2026-02-22T21:13:33.845 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:33.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672975 2026-02-22T21:13:33.847 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2026-02-22T21:13:33.847 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 42949672975 2026-02-22T21:13:33.847 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:13:34.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672976 -lt 42949672975 2026-02-22T21:13:34.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:34.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280514 2026-02-22T21:13:34.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:34.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:13:34.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280514 2026-02-22T21:13:34.010 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:34.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280514 2026-02-22T21:13:34.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280514' 2026-02-22T21:13:34.011 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 94489280514 2026-02-22T21:13:34.012 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:13:34.181 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280515 -lt 94489280514 2026-02-22T21:13:34.182 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:13:34.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:13:34.183 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:13:34.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:13:34.374 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:13:34.374 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:13:34.374 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:13:34.374 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:13:34.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:13:34.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:13:34.375 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:13:34.531 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:13:34.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:13:34.531 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:13:34.532 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:13:34.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:13:34.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:13:34.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:13:34.733 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:90: TEST_scrub_test: rm -f /tmp/tmp.crXaH5s3fo 2026-02-22T21:13:34.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:92: TEST_scrub_test: local pgid=1.0 2026-02-22T21:13:34.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:93: TEST_scrub_test: pg_deep_scrub 1.0 2026-02-22T21:13:34.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=1.0 2026-02-22T21:13:34.734 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 1.0 2026-02-22T21:13:34.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-22T21:13:34.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T21:13:34.735 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:13:34.735 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:13:34.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:13:34.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:13:34.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:13:35.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T21:13:35.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T21:13:35.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T21:13:35.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T21:13:35.025 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:13:35.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:13:35.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:13:35.179 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:13:35.179 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:13:35.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:13:35.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:35.179 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:13:35.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836503 2026-02-22T21:13:35.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836503 2026-02-22T21:13:35.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503' 2026-02-22T21:13:35.251 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:35.251 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:13:35.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672979 2026-02-22T21:13:35.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672979 2026-02-22T21:13:35.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672979' 2026-02-22T21:13:35.323 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:35.324 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:13:35.393 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280518 2026-02-22T21:13:35.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280518 2026-02-22T21:13:35.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836503 1-42949672979 2-94489280518' 2026-02-22T21:13:35.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:35.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836503 2026-02-22T21:13:35.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:35.396 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:13:35.396 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836503 2026-02-22T21:13:35.397 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:35.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836503 2026-02-22T21:13:35.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2026-02-22T21:13:35.398 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836503 2026-02-22T21:13:35.399 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:35.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836503 2026-02-22T21:13:35.567 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:13:36.568 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:13:36.569 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:36.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836502 -lt 21474836503 2026-02-22T21:13:36.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:13:37.726 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-22T21:13:37.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:37.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836505 -lt 21474836503 2026-02-22T21:13:37.889 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:37.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672979 2026-02-22T21:13:37.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:37.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:13:37.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672979 2026-02-22T21:13:37.892 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:37.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672979 2026-02-22T21:13:37.893 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672979 2026-02-22T21:13:37.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2026-02-22T21:13:37.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:13:38.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672981 -lt 42949672979 2026-02-22T21:13:38.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:38.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280518 2026-02-22T21:13:38.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:38.054 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:13:38.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280518 2026-02-22T21:13:38.055 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:38.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280518 2026-02-22T21:13:38.056 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280518' 2026-02-22T21:13:38.057 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 94489280518 2026-02-22T21:13:38.057 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:13:38.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280520 -lt 94489280518 2026-02-22T21:13:38.219 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T21:13:38.219 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 1.0 loop 0 2026-02-22T21:13:38.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-22T21:13:38.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T21:13:38.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-22T21:13:38.220 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T21:13:38.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-22T21:13:38.220 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T21:13:38.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-22T21:13:38.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T21:13:38.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T21:13:38.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T21:13:38.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-22T21:13:38.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:38.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T21:13:38.292 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:38.292 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-22T21:13:38.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-22T21:13:24.081446+0000 2026-02-22T21:13:38.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 1.0 2026-02-22T21:13:38.592 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2026-02-22T21:13:38.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 1.0 2026-02-22T21:13:24.081446+0000 last_deep_scrub_stamp 2026-02-22T21:13:38.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T21:13:38.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T21:13:24.081446+0000 2026-02-22T21:13:38.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-22T21:13:38.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T21:13:38.601 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:13:38.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-22T21:13:38.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:38.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T21:13:38.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:38.602 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-22T21:13:38.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T21:13:24.081446+0000 '>' 2026-02-22T21:13:24.081446+0000 2026-02-22T21:13:38.762 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:13:39.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:13:39.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:13:39.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-22T21:13:39.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:39.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T21:13:39.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:39.765 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-22T21:13:39.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T21:13:38.623890+0000 '>' 2026-02-22T21:13:24.081446+0000 2026-02-22T21:13:39.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T21:13:39.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:95: TEST_scrub_test: ceph pg dump pgs 2026-02-22T21:13:39.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:95: TEST_scrub_test: grep '^1.0' 2026-02-22T21:13:39.922 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:95: TEST_scrub_test: grep -q -- +inconsistent 2026-02-22T21:13:40.070 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:13:40.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:96: TEST_scrub_test: ceph pg 1.0 query 2026-02-22T21:13:40.081 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:96: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2026-02-22T21:13:40.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:96: TEST_scrub_test: test 2 = 2 2026-02-22T21:13:40.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:98: TEST_scrub_test: ceph osd out 1 2026-02-22T21:13:40.358 INFO:tasks.workunit.client.0.trial150.stderr:marked out osd.1. 2026-02-22T21:13:40.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:99: TEST_scrub_test: wait_for_clean 2026-02-22T21:13:40.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:13:40.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:13:40.372 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:13:40.372 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:13:40.373 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:13:40.373 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:13:40.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:13:40.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:13:40.373 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:13:40.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:13:40.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:13:40.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:13:40.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:13:40.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:13:40.493 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:13:40.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:13:40.653 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:13:40.653 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:13:40.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:13:40.653 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:40.653 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:13:40.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836509 2026-02-22T21:13:40.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836509 2026-02-22T21:13:40.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509' 2026-02-22T21:13:40.731 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:40.731 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:13:40.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672986 2026-02-22T21:13:40.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672986 2026-02-22T21:13:40.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672986' 2026-02-22T21:13:40.802 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:40.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:13:40.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280525 2026-02-22T21:13:40.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280525 2026-02-22T21:13:40.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836509 1-42949672986 2-94489280525' 2026-02-22T21:13:40.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:40.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836509 2026-02-22T21:13:40.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:40.876 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:13:40.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836509 2026-02-22T21:13:40.877 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:40.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836509 2026-02-22T21:13:40.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2026-02-22T21:13:40.879 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836509 2026-02-22T21:13:40.879 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:41.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836507 -lt 21474836509 2026-02-22T21:13:41.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:13:42.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:13:42.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:42.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836510 -lt 21474836509 2026-02-22T21:13:42.204 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:42.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672986 2026-02-22T21:13:42.204 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:42.206 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:13:42.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672986 2026-02-22T21:13:42.207 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:42.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672986 2026-02-22T21:13:42.209 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2026-02-22T21:13:42.209 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672986 2026-02-22T21:13:42.209 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:13:42.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672986 -lt 42949672986 2026-02-22T21:13:42.366 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:42.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280525 2026-02-22T21:13:42.367 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:42.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:13:42.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280525 2026-02-22T21:13:42.369 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:42.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280525 2026-02-22T21:13:42.371 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280525' 2026-02-22T21:13:42.371 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 94489280525 2026-02-22T21:13:42.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:13:42.539 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280525 -lt 94489280525 2026-02-22T21:13:42.540 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:13:42.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:13:42.541 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:13:42.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:13:42.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:13:42.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:13:42.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:13:42.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:13:42.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:13:42.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:13:42.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:13:42.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:13:42.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:13:42.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:13:42.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:13:43.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:13:43.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:13:43.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:13:43.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:101: TEST_scrub_test: pg_deep_scrub 1.0 2026-02-22T21:13:43.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1932: pg_deep_scrub: local pgid=1.0 2026-02-22T21:13:43.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: pg_deep_scrub: wait_for_pg_clean 1.0 2026-02-22T21:13:43.079 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-22T21:13:43.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-22T21:13:43.080 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:13:43.080 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:13:43.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:13:43.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:13:43.080 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:13:43.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-22T21:13:43.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-22T21:13:43.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-22T21:13:43.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-22T21:13:43.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:13:43.363 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:13:43.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:13:43.516 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:13:43.517 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:13:43.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:13:43.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:43.517 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:13:43.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836513 2026-02-22T21:13:43.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836513 2026-02-22T21:13:43.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513' 2026-02-22T21:13:43.591 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:43.591 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:13:43.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672989 2026-02-22T21:13:43.662 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672989 2026-02-22T21:13:43.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672989' 2026-02-22T21:13:43.663 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:43.663 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:13:43.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280529 2026-02-22T21:13:43.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280529 2026-02-22T21:13:43.735 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836513 1-42949672989 2-94489280529' 2026-02-22T21:13:43.736 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:43.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836513 2026-02-22T21:13:43.736 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:43.737 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:13:43.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836513 2026-02-22T21:13:43.737 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:43.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836513 2026-02-22T21:13:43.739 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2026-02-22T21:13:43.739 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836513 2026-02-22T21:13:43.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:43.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836513 2026-02-22T21:13:43.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:13:44.906 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:13:44.906 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:45.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836512 -lt 21474836513 2026-02-22T21:13:45.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:13:46.067 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 499 -eq 0 ']' 2026-02-22T21:13:46.068 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:46.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836515 -lt 21474836513 2026-02-22T21:13:46.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:46.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672989 2026-02-22T21:13:46.227 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:46.228 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:13:46.229 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672989 2026-02-22T21:13:46.229 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:46.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672989 2026-02-22T21:13:46.231 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2026-02-22T21:13:46.231 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672989 2026-02-22T21:13:46.232 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:13:46.397 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672991 -lt 42949672989 2026-02-22T21:13:46.398 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:46.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280529 2026-02-22T21:13:46.398 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:46.399 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:13:46.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280529 2026-02-22T21:13:46.400 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:46.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280529 2026-02-22T21:13:46.402 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280529' 2026-02-22T21:13:46.402 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 94489280529 2026-02-22T21:13:46.402 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:13:46.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280530 -lt 94489280529 2026-02-22T21:13:46.569 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-22T21:13:46.569 INFO:tasks.workunit.client.0.trial150.stdout:#---------- 1.0 loop 0 2026-02-22T21:13:46.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-22T21:13:46.570 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-22T21:13:46.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-22T21:13:46.571 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-22T21:13:46.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-22T21:13:46.571 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-22T21:13:46.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean+remapped 2026-02-22T21:13:46.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean+remapped == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-22T21:13:46.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-22T21:13:46.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-22T21:13:46.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-22T21:13:46.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:46.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T21:13:46.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:46.643 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-22T21:13:46.794 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: pg_deep_scrub: local last_scrub=2026-02-22T21:13:38.623890+0000 2026-02-22T21:13:46.795 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: pg_deep_scrub: ceph pg deep-scrub 1.0 2026-02-22T21:13:46.947 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0 on osd.0 to deep-scrub 2026-02-22T21:13:46.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: pg_deep_scrub: wait_for_scrub 1.0 2026-02-22T21:13:38.623890+0000 last_deep_scrub_stamp 2026-02-22T21:13:46.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T21:13:46.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T21:13:38.623890+0000 2026-02-22T21:13:46.957 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_deep_scrub_stamp 2026-02-22T21:13:46.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T21:13:46.958 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:13:46.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-22T21:13:46.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:46.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T21:13:46.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:46.958 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-22T21:13:47.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T21:13:38.623890+0000 '>' 2026-02-22T21:13:38.623890+0000 2026-02-22T21:13:47.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:13:48.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:13:48.115 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:13:48.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-22T21:13:48.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:48.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T21:13:48.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:48.116 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-22T21:13:48.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T21:13:38.623890+0000 '>' 2026-02-22T21:13:38.623890+0000 2026-02-22T21:13:48.275 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:13:49.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:13:49.277 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:13:49.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_deep_scrub_stamp 2026-02-22T21:13:49.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:49.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_deep_scrub_stamp 2026-02-22T21:13:49.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:49.278 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_deep_scrub_stamp' 2026-02-22T21:13:49.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T21:13:47.555264+0000 '>' 2026-02-22T21:13:38.623890+0000 2026-02-22T21:13:49.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T21:13:49.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:103: TEST_scrub_test: ceph pg 1.0 query 2026-02-22T21:13:49.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:103: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2026-02-22T21:13:49.510 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:103: TEST_scrub_test: test 2 = 2 2026-02-22T21:13:49.511 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: ceph pg 1.0 query 2026-02-22T21:13:49.511 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2026-02-22T21:13:49.583 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:104: TEST_scrub_test: test 2 = 2 2026-02-22T21:13:49.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:105: TEST_scrub_test: ceph pg dump pgs 2026-02-22T21:13:49.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:105: TEST_scrub_test: grep '^1.0' 2026-02-22T21:13:49.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:105: TEST_scrub_test: grep -q -- +inconsistent 2026-02-22T21:13:49.733 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:13:49.742 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:107: TEST_scrub_test: ceph osd in 1 2026-02-22T21:13:49.940 INFO:tasks.workunit.client.0.trial150.stderr:marked in osd.1. 2026-02-22T21:13:49.954 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:108: TEST_scrub_test: wait_for_clean 2026-02-22T21:13:49.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:13:49.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:13:49.955 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:13:49.955 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:13:49.956 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:13:49.956 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:13:49.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:13:49.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:13:49.956 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:13:50.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:13:50.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:13:50.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:13:50.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:13:50.061 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:13:50.061 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:13:50.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:13:50.221 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:13:50.221 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:13:50.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:13:50.221 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:50.221 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:13:50.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836521 2026-02-22T21:13:50.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836521 2026-02-22T21:13:50.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836521' 2026-02-22T21:13:50.293 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:50.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:13:50.363 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672997 2026-02-22T21:13:50.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672997 2026-02-22T21:13:50.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836521 1-42949672997' 2026-02-22T21:13:50.364 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:50.364 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:13:50.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280536 2026-02-22T21:13:50.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280536 2026-02-22T21:13:50.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836521 1-42949672997 2-94489280536' 2026-02-22T21:13:50.434 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:50.435 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836521 2026-02-22T21:13:50.435 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:50.436 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:13:50.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836521 2026-02-22T21:13:50.437 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:50.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836521 2026-02-22T21:13:50.439 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2026-02-22T21:13:50.439 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836521 2026-02-22T21:13:50.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:50.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836519 -lt 21474836521 2026-02-22T21:13:50.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:13:51.594 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:13:51.594 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:51.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836522 -lt 21474836521 2026-02-22T21:13:51.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:51.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672997 2026-02-22T21:13:51.752 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:51.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:13:51.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672997 2026-02-22T21:13:51.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:51.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672997 2026-02-22T21:13:51.756 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2026-02-22T21:13:51.756 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672997 2026-02-22T21:13:51.756 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:13:51.927 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672998 -lt 42949672997 2026-02-22T21:13:51.928 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:51.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280536 2026-02-22T21:13:51.928 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:51.930 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:13:51.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280536 2026-02-22T21:13:51.931 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:51.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280536 2026-02-22T21:13:51.932 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 94489280536 2026-02-22T21:13:51.932 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280536' 2026-02-22T21:13:51.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:13:52.101 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280537 -lt 94489280536 2026-02-22T21:13:52.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:13:52.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:13:52.102 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:13:52.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:13:52.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:13:52.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:13:52.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:13:52.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:13:52.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:13:52.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:13:52.291 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:13:52.450 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:13:52.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:13:52.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:13:52.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:13:52.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:13:52.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:13:52.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:13:52.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:110: TEST_scrub_test: repair 1.0 2026-02-22T21:13:52.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1887: repair: local pgid=1.0 2026-02-22T21:13:52.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: get_last_scrub_stamp 1.0 2026-02-22T21:13:52.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:52.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:52.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:52.645 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:52.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: repair: local last_scrub=2026-02-22T21:13:47.555264+0000 2026-02-22T21:13:52.805 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: repair: ceph pg repair 1.0 2026-02-22T21:13:52.956 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0 on osd.1 to repair 2026-02-22T21:13:52.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: repair: wait_for_scrub 1.0 2026-02-22T21:13:47.555264+0000 2026-02-22T21:13:52.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-22T21:13:52.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-22T21:13:47.555264+0000 2026-02-22T21:13:52.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-22T21:13:52.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-22T21:13:52.967 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:13:52.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:13:52.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:52.968 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:52.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:52.969 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:53.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T21:13:47.555264+0000 '>' 2026-02-22T21:13:47.555264+0000 2026-02-22T21:13:53.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:13:54.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:13:54.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:13:54.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:13:54.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:54.129 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:54.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:54.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:54.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T21:13:47.555264+0000 '>' 2026-02-22T21:13:47.555264+0000 2026-02-22T21:13:54.291 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-22T21:13:55.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-22T21:13:55.292 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 500 )) 2026-02-22T21:13:55.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-22T21:13:55.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-22T21:13:55.293 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-22T21:13:55.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-22T21:13:55.294 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-22T21:13:55.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-22T21:13:54.539173+0000 '>' 2026-02-22T21:13:47.555264+0000 2026-02-22T21:13:55.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-22T21:13:55.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:111: TEST_scrub_test: wait_for_clean 2026-02-22T21:13:55.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:13:55.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:13:55.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:13:55.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:13:55.454 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:13:55.455 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:13:55.455 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:13:55.455 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:13:55.455 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:13:55.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:13:55.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:13:55.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:13:55.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:13:55.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:13:55.563 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:13:55.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:13:55.725 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:13:55.725 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:13:55.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:13:55.725 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:55.726 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:13:55.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836527 2026-02-22T21:13:55.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836527 2026-02-22T21:13:55.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836527' 2026-02-22T21:13:55.798 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:55.798 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:13:55.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673004 2026-02-22T21:13:55.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673004 2026-02-22T21:13:55.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836527 1-42949673004' 2026-02-22T21:13:55.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:55.871 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:13:55.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280543 2026-02-22T21:13:55.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280543 2026-02-22T21:13:55.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836527 1-42949673004 2-94489280543' 2026-02-22T21:13:55.944 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:55.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836527 2026-02-22T21:13:55.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:55.946 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:13:55.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836527 2026-02-22T21:13:55.947 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:55.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836527 2026-02-22T21:13:55.948 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2026-02-22T21:13:55.948 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836527 2026-02-22T21:13:55.949 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:56.107 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836526 -lt 21474836527 2026-02-22T21:13:56.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:13:57.109 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 500 -eq 0 ']' 2026-02-22T21:13:57.109 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:57.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836529 -lt 21474836527 2026-02-22T21:13:57.267 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:57.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673004 2026-02-22T21:13:57.268 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:57.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:13:57.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673004 2026-02-22T21:13:57.270 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673004 2026-02-22T21:13:57.272 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2026-02-22T21:13:57.272 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949673004 2026-02-22T21:13:57.273 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:13:57.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673005 -lt 42949673004 2026-02-22T21:13:57.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:57.438 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280543 2026-02-22T21:13:57.439 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:57.440 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:13:57.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280543 2026-02-22T21:13:57.441 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:57.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280543 2026-02-22T21:13:57.442 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 94489280543 2026-02-22T21:13:57.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280543' 2026-02-22T21:13:57.443 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:13:57.610 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280544 -lt 94489280543 2026-02-22T21:13:57.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:13:57.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:13:57.611 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:13:57.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:13:57.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:13:57.803 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:13:57.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:13:57.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:13:57.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:13:57.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:13:57.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:13:57.960 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:13:57.960 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:13:57.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:13:57.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:13:58.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:13:58.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:13:58.158 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:13:58.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:114: TEST_scrub_test: ceph pg 1.0 query 2026-02-22T21:13:58.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:114: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2026-02-22T21:13:58.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:114: TEST_scrub_test: test 2 = 2 2026-02-22T21:13:58.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: TEST_scrub_test: ceph pg dump pgs 2026-02-22T21:13:58.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: TEST_scrub_test: grep '^1.0' 2026-02-22T21:13:58.232 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:115: TEST_scrub_test: grep -vq -- +inconsistent 2026-02-22T21:13:58.383 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:13:58.394 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:117: TEST_scrub_test: ceph osd out 1 2026-02-22T21:13:58.594 INFO:tasks.workunit.client.0.trial150.stderr:marked out osd.1. 2026-02-22T21:13:58.606 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:118: TEST_scrub_test: wait_for_clean 2026-02-22T21:13:58.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:13:58.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:13:58.607 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:13:58.607 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:13:58.607 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:13:58.607 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:13:58.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:13:58.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:13:58.608 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:13:58.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:13:58.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:13:58.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:13:58.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:13:58.727 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=500 2026-02-22T21:13:58.727 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:13:58.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:13:58.886 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:13:58.886 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:13:58.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:13:58.886 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:58.886 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:13:58.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836532 2026-02-22T21:13:58.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836532 2026-02-22T21:13:58.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532' 2026-02-22T21:13:58.961 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:58.961 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:13:59.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949673008 2026-02-22T21:13:59.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949673008 2026-02-22T21:13:59.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532 1-42949673008' 2026-02-22T21:13:59.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:13:59.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:13:59.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=94489280547 2026-02-22T21:13:59.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 94489280547 2026-02-22T21:13:59.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836532 1-42949673008 2-94489280547' 2026-02-22T21:13:59.103 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:59.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836532 2026-02-22T21:13:59.104 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:59.105 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:13:59.106 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836532 2026-02-22T21:13:59.106 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:59.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836532 2026-02-22T21:13:59.108 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2026-02-22T21:13:59.108 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836532 2026-02-22T21:13:59.108 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:13:59.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836532 -lt 21474836532 2026-02-22T21:13:59.276 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:59.276 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949673008 2026-02-22T21:13:59.277 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:59.278 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:13:59.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949673008 2026-02-22T21:13:59.279 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:59.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949673008 2026-02-22T21:13:59.280 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949673008 2026-02-22T21:13:59.280 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2026-02-22T21:13:59.281 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:13:59.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949673008 -lt 42949673008 2026-02-22T21:13:59.441 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:13:59.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-94489280547 2026-02-22T21:13:59.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:13:59.443 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:13:59.444 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-94489280547 2026-02-22T21:13:59.444 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:13:59.445 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=94489280547 2026-02-22T21:13:59.446 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 94489280547' 2026-02-22T21:13:59.446 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 94489280547 2026-02-22T21:13:59.446 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:13:59.608 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 94489280547 -lt 94489280547 2026-02-22T21:13:59.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:13:59.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:13:59.609 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:13:59.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:13:59.806 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:13:59.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:13:59.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:13:59.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:13:59.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:13:59.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:13:59.807 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:13:59.972 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:13:59.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:13:59.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:13:59.973 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:00.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:14:00.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:14:00.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:14:00.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:120: TEST_scrub_test: ceph pg 1.0 query 2026-02-22T21:14:00.163 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:120: TEST_scrub_test: jq .info.stats.stat_sum.num_scrub_errors 2026-02-22T21:14:00.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:120: TEST_scrub_test: test 0 = 0 2026-02-22T21:14:00.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: TEST_scrub_test: ceph pg 1.0 query 2026-02-22T21:14:00.234 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: TEST_scrub_test: jq '.peer_info[0].stats.stat_sum.num_scrub_errors' 2026-02-22T21:14:00.308 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:121: TEST_scrub_test: test 0 = 0 2026-02-22T21:14:00.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: TEST_scrub_test: ceph pg 1.0 query 2026-02-22T21:14:00.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: TEST_scrub_test: jq '.peer_info[1].stats.stat_sum.num_scrub_errors' 2026-02-22T21:14:00.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:122: TEST_scrub_test: test 0 = 0 2026-02-22T21:14:00.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:123: TEST_scrub_test: ceph pg dump pgs 2026-02-22T21:14:00.381 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:123: TEST_scrub_test: grep '^1.0' 2026-02-22T21:14:00.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:123: TEST_scrub_test: grep -vq -- +inconsistent 2026-02-22T21:14:00.525 INFO:tasks.workunit.client.0.trial150.stderr:dumped pgs 2026-02-22T21:14:00.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:124: TEST_scrub_test: perf_counters td/osd-scrub-test 3 2026-02-22T21:14:00.535 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:43: perf_counters: local dir=td/osd-scrub-test 2026-02-22T21:14:00.536 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:44: perf_counters: local OSDS=3 2026-02-22T21:14:00.536 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: expr 3 - 1 2026-02-22T21:14:00.537 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: seq 0 2 2026-02-22T21:14:00.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:14:00.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.0 counter dump 2026-02-22T21:14:00.538 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 2, 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:14:00.615 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:14:00.616 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:14:00.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:14:00.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.1 counter dump 2026-02-22T21:14:00.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:14:00.687 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 6, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:14:00.688 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:14:00.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:45: perf_counters: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-22T21:14:00.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: ceph tell osd.2 counter dump 2026-02-22T21:14:00.689 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:47: perf_counters: jq 'with_entries(select(.key | startswith("osd_scrub")))' 2026-02-22T21:14:00.758 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:14:00.758 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_ec": [ 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.759 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_dp_repl": [ 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: "level": "deep", 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.760 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_ec": [ 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "ec" 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:14:00.761 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.762 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.762 INFO:tasks.workunit.client.0.trial150.stdout: ], 2026-02-22T21:14:00.762 INFO:tasks.workunit.client.0.trial150.stdout: "osd_scrub_sh_repl": [ 2026-02-22T21:14:00.762 INFO:tasks.workunit.client.0.trial150.stdout: { 2026-02-22T21:14:00.762 INFO:tasks.workunit.client.0.trial150.stdout: "labels": { 2026-02-22T21:14:00.762 INFO:tasks.workunit.client.0.trial150.stdout: "level": "shallow", 2026-02-22T21:14:00.762 INFO:tasks.workunit.client.0.trial150.stdout: "pooltype": "replicated" 2026-02-22T21:14:00.762 INFO:tasks.workunit.client.0.trial150.stdout: }, 2026-02-22T21:14:00.762 INFO:tasks.workunit.client.0.trial150.stdout: "counters": { 2026-02-22T21:14:00.762 INFO:tasks.workunit.client.0.trial150.stdout: "preemptions": 0, 2026-02-22T21:14:00.762 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_selected": 0, 2026-02-22T21:14:00.762 INFO:tasks.workunit.client.0.trial150.stdout: "chunk_busy": 0, 2026-02-22T21:14:00.763 INFO:tasks.workunit.client.0.trial150.stdout: "locked_object": 0, 2026-02-22T21:14:00.763 INFO:tasks.workunit.client.0.trial150.stdout: "write_blocked_by_scrub": 0 2026-02-22T21:14:00.763 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.763 INFO:tasks.workunit.client.0.trial150.stdout: } 2026-02-22T21:14:00.763 INFO:tasks.workunit.client.0.trial150.stdout: ] 2026-02-22T21:14:00.763 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:14:00.763 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Teardown Test TEST_scrub_test ------------------ 2026-02-22T21:14:00.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:36: run: echo '-------------- Teardown Test TEST_scrub_test ------------------' 2026-02-22T21:14:00.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:37: run: teardown td/osd-scrub-test 2026-02-22T21:14:00.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:14:00.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:14:00.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:14:00.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:14:00.764 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:14:00.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:14:00.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:14:00.765 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:14:00.869 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:14:00.869 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:14:00.870 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:14:00.870 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:14:00.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:14:00.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:14:00.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:14:00.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:14:00.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:14:00.874 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:14:00.874 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:14:00.875 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:14:00.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:14:00.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:14:00.889 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:14:00.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:00.890 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:14:00.890 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:14:00.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:14:00.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:14:00.891 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-scrub-test.sh:38: run: echo '-------------- Complete Test TEST_scrub_test ------------------' 2026-02-22T21:14:00.892 INFO:tasks.workunit.client.0.trial150.stdout:-------------- Complete Test TEST_scrub_test ------------------ 2026-02-22T21:14:00.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-22T21:14:00.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-scrub-test 0 2026-02-22T21:14:00.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-scrub-test 2026-02-22T21:14:00.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-22T21:14:00.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-scrub-test KILL 2026-02-22T21:14:00.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:14:00.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:14:00.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:14:00.893 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:14:00.894 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:14:00.895 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:14:00.895 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:14:00.896 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:14:00.896 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:14:00.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:14:00.897 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:14:00.897 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:14:00.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:14:00.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:14:00.899 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:14:00.899 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:14:00.901 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:14:00.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T21:14:00.902 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-scrub-test 2026-02-22T21:14:00.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:14:00.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:00.903 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.516403 2026-02-22T21:14:00.903 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.516403 2026-02-22T21:14:00.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:14:00.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:14:00.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-22T21:14:00.905 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T21:14:00.905 DEBUG:teuthology.orchestra.run.trial150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T21:14:00.960 INFO:tasks.workunit:Running workunit scrub/osd-unexpected-clone.sh... 2026-02-22T21:14:00.960 DEBUG:teuthology.orchestra.run.trial150:workunit test scrub/osd-unexpected-clone.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a5c3084f688b1f6eff223b8d259a8316f9844c8e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh 2026-02-22T21:14:01.012 INFO:tasks.workunit.client.0.trial150.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/osd-unexpected-clone 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:22: run: local dir=td/osd-unexpected-clone 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:23: run: shift 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:25: run: export CEPH_MON=127.0.0.1:7144 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:25: run: CEPH_MON=127.0.0.1:7144 2026-02-22T21:14:01.015 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:26: run: export CEPH_ARGS 2026-02-22T21:14:01.016 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:27: run: uuidgen 2026-02-22T21:14:01.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:27: run: CEPH_ARGS+='--fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none ' 2026-02-22T21:14:01.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7144 ' 2026-02-22T21:14:01.017 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:30: run: export -n CEPH_CLI_TEST_DUP_COMMAND 2026-02-22T21:14:01.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: set 2026-02-22T21:14:01.017 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-22T21:14:01.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:31: run: local funcs=TEST_recover_unexpected 2026-02-22T21:14:01.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:32: run: for func in $funcs 2026-02-22T21:14:01.019 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:33: run: setup td/osd-unexpected-clone 2026-02-22T21:14:01.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-unexpected-clone 2026-02-22T21:14:01.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-unexpected-clone 2026-02-22T21:14:01.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-unexpected-clone 2026-02-22T21:14:01.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:14:01.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-unexpected-clone KILL 2026-02-22T21:14:01.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:14:01.020 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:14:01.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:14:01.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:14:01.020 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:14:01.022 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:14:01.022 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:14:01.023 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:14:01.023 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:14:01.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:14:01.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:14:01.025 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:14:01.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:14:01.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:14:01.026 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:14:01.026 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:14:01.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:14:01.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:14:01.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-unexpected-clone 2026-02-22T21:14:01.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:14:01.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:01.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:01.031 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.662307 2026-02-22T21:14:01.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:14:01.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:14:01.032 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-unexpected-clone 2026-02-22T21:14:01.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-22T21:14:01.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:01.033 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:01.034 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.662307 2026-02-22T21:14:01.035 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-22T21:14:01.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-22T21:14:01.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-22T21:14:01.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-22T21:14:01.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-unexpected-clone 1' TERM HUP INT 2026-02-22T21:14:01.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:34: run: TEST_recover_unexpected td/osd-unexpected-clone 2026-02-22T21:14:01.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:40: TEST_recover_unexpected: local dir=td/osd-unexpected-clone 2026-02-22T21:14:01.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:42: TEST_recover_unexpected: run_mon td/osd-unexpected-clone a 2026-02-22T21:14:01.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-unexpected-clone 2026-02-22T21:14:01.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-22T21:14:01.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-22T21:14:01.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-22T21:14:01.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-unexpected-clone/a 2026-02-22T21:14:01.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-unexpected-clone/a --run-dir=td/osd-unexpected-clone 2026-02-22T21:14:01.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-22T21:14:01.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:14:01.059 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:14:01.060 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:14:01.060 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:01.060 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:01.060 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:01.060 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-unexpected-clone/a '--log-file=td/osd-unexpected-clone/$name.log' '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --mon-cluster-log-file=td/osd-unexpected-clone/log --run-dir=td/osd-unexpected-clone '--pid-file=td/osd-unexpected-clone/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-22T21:14:01.089 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-22T21:14:01.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-22T21:14:01.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:14:01.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:14:01.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-22T21:14:01.091 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-22T21:14:01.091 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:14:01.092 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:14:01.092 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:14:01.092 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:14:01.092 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:01.092 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:01.092 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.662307/ceph-mon.a.asok 2026-02-22T21:14:01.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:14:01.093 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.662307/ceph-mon.a.asok config get fsid 2026-02-22T21:14:01.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-22T21:14:01.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-22T21:14:01.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-22T21:14:01.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-22T21:14:01.158 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-22T21:14:01.158 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-22T21:14:01.158 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-22T21:14:01.158 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-22T21:14:01.159 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-22T21:14:01.159 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:01.159 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:01.159 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.662307/ceph-mon.a.asok 2026-02-22T21:14:01.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-22T21:14:01.159 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.662307/ceph-mon.a.asok config get mon_host 2026-02-22T21:14:01.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:43: TEST_recover_unexpected: run_mgr td/osd-unexpected-clone x 2026-02-22T21:14:01.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-unexpected-clone 2026-02-22T21:14:01.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-22T21:14:01.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-22T21:14:01.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-22T21:14:01.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-unexpected-clone/x 2026-02-22T21:14:01.217 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-22T21:14:01.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-22T21:14:01.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:14:01.309 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:14:01.310 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:14:01.310 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:01.310 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:01.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:01.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:14:01.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-unexpected-clone/x '--log-file=td/osd-unexpected-clone/$name.log' '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --run-dir=td/osd-unexpected-clone '--pid-file=td/osd-unexpected-clone/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-22T21:14:01.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:44: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 0 2026-02-22T21:14:01.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-unexpected-clone 2026-02-22T21:14:01.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:14:01.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-22T21:14:01.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:14:01.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-unexpected-clone/0 2026-02-22T21:14:01.328 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/0' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/0/journal' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-22T21:14:01.329 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:14:01.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:14:01.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:14:01.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:14:01.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:14:01.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T21:14:01.330 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-unexpected-clone/0 2026-02-22T21:14:01.331 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:14:01.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=55b48909-2275-46c1-a3f4-0be03759837d 2026-02-22T21:14:01.332 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 55b48909-2275-46c1-a3f4-0be03759837d' 2026-02-22T21:14:01.332 INFO:tasks.workunit.client.0.trial150.stdout:add osd0 55b48909-2275-46c1-a3f4-0be03759837d 2026-02-22T21:14:01.332 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:14:01.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCZcZtp6RYYFBAAeFcyDpDgvGS4rCWGdtPlEg== 2026-02-22T21:14:01.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCZcZtp6RYYFBAAeFcyDpDgvGS4rCWGdtPlEg=="}' 2026-02-22T21:14:01.342 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 55b48909-2275-46c1-a3f4-0be03759837d -i td/osd-unexpected-clone/0/new.json 2026-02-22T21:14:01.429 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:14:01.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-unexpected-clone/0/new.json 2026-02-22T21:14:01.438 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/0 --osd-journal=td/osd-unexpected-clone/0/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZcZtp6RYYFBAAeFcyDpDgvGS4rCWGdtPlEg== --osd-uuid 55b48909-2275-46c1-a3f4-0be03759837d 2026-02-22T21:14:01.454 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:01.447+0000 7f6559029a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:01.455 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:01.447+0000 7f6559029a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:01.457 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:01.451+0000 7f6559029a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:01.457 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:01.451+0000 7f6559029a40 -1 bdev(0x559169f18c00 td/osd-unexpected-clone/0/block) open stat got: (1) Operation not permitted 2026-02-22T21:14:01.457 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:01.451+0000 7f6559029a40 -1 bluestore(td/osd-unexpected-clone/0) _read_fsid unparsable uuid 2026-02-22T21:14:02.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-unexpected-clone/0/keyring 2026-02-22T21:14:02.207 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:14:02.208 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-22T21:14:02.209 INFO:tasks.workunit.client.0.trial150.stdout:adding osd0 key to auth repository 2026-02-22T21:14:02.210 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-unexpected-clone/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:14:02.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-22T21:14:02.403 INFO:tasks.workunit.client.0.trial150.stdout:start osd.0 2026-02-22T21:14:02.403 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/0 --osd-journal=td/osd-unexpected-clone/0/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:14:02.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:14:02.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:14:02.404 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:14:02.427 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:02.419+0000 7f3849c0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:02.428 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:02.423+0000 7f3849c0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:02.429 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:02.423+0000 7f3849c0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:02.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-22T21:14:02.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:14:02.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-22T21:14:02.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:14:02.562 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:14:02.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:02.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:14:02.563 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:14:02.563 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:02.564 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:14:02.721 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:14:02.794 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:02.787+0000 7f3849c0ea40 -1 Falling back to public interface 2026-02-22T21:14:03.367 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:03.359+0000 7f3849c0ea40 -1 osd.0 0 log_to_monitors true 2026-02-22T21:14:03.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:14:03.723 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:03.723 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:14:03.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:14:03.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:03.724 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:14:03.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:14:04.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:14:04.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:04.878 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:14:04.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:14:04.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:04.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-22T21:14:05.038 INFO:tasks.workunit.client.0.trial150.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3461833219,v1:127.0.0.1:6803/3461833219] [v2:127.0.0.1:6804/3461833219,v1:127.0.0.1:6805/3461833219] exists,up 55b48909-2275-46c1-a3f4-0be03759837d 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:45: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 1 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-unexpected-clone 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-unexpected-clone/1 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:05.039 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T21:14:05.040 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-unexpected-clone/1 2026-02-22T21:14:05.041 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:14:05.041 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b649b842-0a09-47b4-a847-12ce2772be45 2026-02-22T21:14:05.042 INFO:tasks.workunit.client.0.trial150.stdout:add osd1 b649b842-0a09-47b4-a847-12ce2772be45 2026-02-22T21:14:05.042 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 b649b842-0a09-47b4-a847-12ce2772be45' 2026-02-22T21:14:05.042 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:14:05.052 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCdcZtptEbWAhAAslLLzzFJIwTNcDsUGErweQ== 2026-02-22T21:14:05.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCdcZtptEbWAhAAslLLzzFJIwTNcDsUGErweQ=="}' 2026-02-22T21:14:05.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b649b842-0a09-47b4-a847-12ce2772be45 -i td/osd-unexpected-clone/1/new.json 2026-02-22T21:14:05.203 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:14:05.214 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-unexpected-clone/1/new.json 2026-02-22T21:14:05.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdcZtptEbWAhAAslLLzzFJIwTNcDsUGErweQ== --osd-uuid b649b842-0a09-47b4-a847-12ce2772be45 2026-02-22T21:14:05.238 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:05.231+0000 7f79960b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:05.239 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:05.231+0000 7f79960b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:05.240 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:05.235+0000 7f79960b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:05.240 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:05.235+0000 7f79960b0a40 -1 bdev(0x559ee5c92c00 td/osd-unexpected-clone/1/block) open stat got: (1) Operation not permitted 2026-02-22T21:14:05.241 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:05.235+0000 7f79960b0a40 -1 bluestore(td/osd-unexpected-clone/1) _read_fsid unparsable uuid 2026-02-22T21:14:05.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-unexpected-clone/1/keyring 2026-02-22T21:14:05.904 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:14:05.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-22T21:14:05.905 INFO:tasks.workunit.client.0.trial150.stdout:adding osd1 key to auth repository 2026-02-22T21:14:05.905 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-unexpected-clone/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:14:06.110 INFO:tasks.workunit.client.0.trial150.stdout:start osd.1 2026-02-22T21:14:06.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-22T21:14:06.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:14:06.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:14:06.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:14:06.111 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:14:06.126 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:06.119+0000 7f0a1bda1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:06.127 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:06.119+0000 7f0a1bda1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:06.128 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:06.123+0000 7f0a1bda1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:06.269 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-22T21:14:06.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:14:06.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T21:14:06.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:14:06.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:14:06.270 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:06.270 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:14:06.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:14:06.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:06.271 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:14:06.429 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:14:06.482 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:06.475+0000 7f0a1bda1a40 -1 Falling back to public interface 2026-02-22T21:14:07.111 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:07.103+0000 7f0a1bda1a40 -1 osd.1 0 log_to_monitors true 2026-02-22T21:14:07.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:14:07.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:07.430 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:14:07.430 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:14:07.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:07.431 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:14:07.584 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:14:08.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:14:08.586 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:08.587 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:14:08.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:14:08.587 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:08.588 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:14:08.746 INFO:tasks.workunit.client.0.trial150.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/416797008,v1:127.0.0.1:6811/416797008] [v2:127.0.0.1:6812/416797008,v1:127.0.0.1:6813/416797008] exists,up b649b842-0a09-47b4-a847-12ce2772be45 2026-02-22T21:14:08.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:14:08.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:14:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:14:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:46: TEST_recover_unexpected: run_osd td/osd-unexpected-clone 2 2026-02-22T21:14:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-unexpected-clone 2026-02-22T21:14:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-22T21:14:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-22T21:14:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-22T21:14:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-unexpected-clone/2 2026-02-22T21:14:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-22T21:14:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:14:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:14:08.750 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:08.751 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:14:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-22T21:14:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-22T21:14:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-22T21:14:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-22T21:14:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:14:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:14:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:14:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:14:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-22T21:14:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-22T21:14:08.752 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-unexpected-clone/2 2026-02-22T21:14:08.753 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-22T21:14:08.753 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d0b51f2f-9685-4c8e-86d4-49fad0ce131a 2026-02-22T21:14:08.753 INFO:tasks.workunit.client.0.trial150.stdout:add osd2 d0b51f2f-9685-4c8e-86d4-49fad0ce131a 2026-02-22T21:14:08.754 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 d0b51f2f-9685-4c8e-86d4-49fad0ce131a' 2026-02-22T21:14:08.754 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-22T21:14:08.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCgcZtpWrIvLRAAJO7A4eQAofTaaopewCivRQ== 2026-02-22T21:14:08.763 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCgcZtpWrIvLRAAJO7A4eQAofTaaopewCivRQ=="}' 2026-02-22T21:14:08.764 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d0b51f2f-9685-4c8e-86d4-49fad0ce131a -i td/osd-unexpected-clone/2/new.json 2026-02-22T21:14:08.923 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:14:08.934 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-unexpected-clone/2/new.json 2026-02-22T21:14:08.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgcZtpWrIvLRAAJO7A4eQAofTaaopewCivRQ== --osd-uuid d0b51f2f-9685-4c8e-86d4-49fad0ce131a 2026-02-22T21:14:08.958 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:08.951+0000 7fc158bb0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:08.960 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:08.951+0000 7fc158bb0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:08.961 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:08.955+0000 7fc158bb0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:08.962 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:08.955+0000 7fc158bb0a40 -1 bdev(0x55f85c4d0c00 td/osd-unexpected-clone/2/block) open stat got: (1) Operation not permitted 2026-02-22T21:14:08.962 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:08.955+0000 7fc158bb0a40 -1 bluestore(td/osd-unexpected-clone/2) _read_fsid unparsable uuid 2026-02-22T21:14:09.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-unexpected-clone/2/keyring 2026-02-22T21:14:09.572 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-22T21:14:09.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-22T21:14:09.573 INFO:tasks.workunit.client.0.trial150.stdout:adding osd2 key to auth repository 2026-02-22T21:14:09.573 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-unexpected-clone/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-22T21:14:09.777 INFO:tasks.workunit.client.0.trial150.stdout:start osd.2 2026-02-22T21:14:09.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-22T21:14:09.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:14:09.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-22T21:14:09.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-22T21:14:09.778 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-22T21:14:09.793 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:09.787+0000 7f1b4ef94a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:09.795 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:09.787+0000 7f1b4ef94a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:09.795 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:09.787+0000 7f1b4ef94a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:09.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-22T21:14:09.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:14:09.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T21:14:09.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:14:09.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:14:09.938 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:09.938 INFO:tasks.workunit.client.0.trial150.stdout:0 2026-02-22T21:14:09.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:14:09.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:09.939 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:14:10.094 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:14:10.234 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:10.227+0000 7f1b4ef94a40 -1 Falling back to public interface 2026-02-22T21:14:10.799 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:10.791+0000 7f1b4ef94a40 -1 osd.2 0 log_to_monitors true 2026-02-22T21:14:11.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:14:11.096 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:11.097 INFO:tasks.workunit.client.0.trial150.stdout:1 2026-02-22T21:14:11.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:14:11.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:11.098 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:14:11.255 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:14:12.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:14:12.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:12.257 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-22T21:14:12.258 INFO:tasks.workunit.client.0.trial150.stdout:2 2026-02-22T21:14:12.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:12.259 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:14:12.412 INFO:tasks.workunit.client.0.trial150.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1333716552,v1:127.0.0.1:6819/1333716552] [v2:127.0.0.1:6820/1333716552,v1:127.0.0.1:6821/1333716552] exists,up d0b51f2f-9685-4c8e-86d4-49fad0ce131a 2026-02-22T21:14:12.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:14:12.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:14:12.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:14:12.412 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:48: TEST_recover_unexpected: ceph osd pool create foo 1 2026-02-22T21:14:12.584 INFO:tasks.workunit.client.0.trial150.stderr:pool 'foo' created 2026-02-22T21:14:12.593 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:49: TEST_recover_unexpected: rados -p foo put foo /etc/passwd 2026-02-22T21:14:13.738 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:50: TEST_recover_unexpected: rados -p foo mksnap snap 2026-02-22T21:14:13.783 INFO:tasks.workunit.client.0.trial150.stdout:created pool foo snap snap 2026-02-22T21:14:13.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:51: TEST_recover_unexpected: rados -p foo put foo /etc/group 2026-02-22T21:14:13.803 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:53: TEST_recover_unexpected: wait_for_clean 2026-02-22T21:14:13.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:14:13.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:14:13.804 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:14:13.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:14:13.804 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:14:13.804 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:14:13.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:14:13.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:14:13.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:14:13.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:14:13.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:14:13.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:14:13.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:14:13.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T21:14:13.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:14:14.075 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:14:14.076 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:14:14.076 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:14:14.076 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:14:14.077 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:14.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:14:14.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-22T21:14:14.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-22T21:14:14.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483' 2026-02-22T21:14:14.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:14.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:14:14.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672963 2026-02-22T21:14:14.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672963 2026-02-22T21:14:14.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-22T21:14:14.216 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:14.216 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:14:14.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509442 2026-02-22T21:14:14.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509442 2026-02-22T21:14:14.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836483 1-42949672963 2-64424509442' 2026-02-22T21:14:14.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:14.286 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-22T21:14:14.286 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:14.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:14:14.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836483 2026-02-22T21:14:14.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:14.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836483 2026-02-22T21:14:14.290 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-22T21:14:14.290 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.0 seq 21474836483 2026-02-22T21:14:14.290 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:14.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T21:14:14.458 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:14:15.459 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:14:15.459 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:15.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836482 -lt 21474836483 2026-02-22T21:14:15.617 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:14:16.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T21:14:16.619 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:16.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-22T21:14:16.780 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:16.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672963 2026-02-22T21:14:16.781 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:16.782 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:14:16.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-42949672963 2026-02-22T21:14:16.783 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:16.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=42949672963 2026-02-22T21:14:16.785 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.1 seq 42949672963 2026-02-22T21:14:16.785 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-22T21:14:16.785 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:14:16.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-22T21:14:16.941 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:16.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509442 2026-02-22T21:14:16.942 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:16.943 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:14:16.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509442 2026-02-22T21:14:16.944 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:16.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509442 2026-02-22T21:14:16.945 INFO:tasks.workunit.client.0.trial150.stdout:waiting osd.2 seq 64424509442 2026-02-22T21:14:16.945 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509442' 2026-02-22T21:14:16.946 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:14:17.113 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509442 -lt 64424509442 2026-02-22T21:14:17.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:14:17.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:17.114 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:17.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:14:17.310 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:14:17.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:14:17.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:14:17.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:14:17.310 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:14:17.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:14:17.311 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:14:17.468 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:14:17.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:14:17.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:17.468 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:17.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:14:17.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:14:17.659 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:14:17.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:55: TEST_recover_unexpected: get_primary foo foo 2026-02-22T21:14:17.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=foo 2026-02-22T21:14:17.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=foo 2026-02-22T21:14:17.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map foo foo 2026-02-22T21:14:17.660 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-22T21:14:17.822 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:55: TEST_recover_unexpected: local osd=1 2026-02-22T21:14:17.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 1 --op list foo 2026-02-22T21:14:17.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: grep snapid.:1 2026-02-22T21:14:17.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-22T21:14:17.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T21:14:17.825 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T21:14:17.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T21:14:17.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 1 --op list foo 2026-02-22T21:14:17.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-22T21:14:17.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T21:14:17.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T21:14:17.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T21:14:17.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.1 2026-02-22T21:14:17.826 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:14:17.826 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:14:17.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:14:17.826 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:14:17.827 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:14:18.131 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:14:18.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 1 --op list foo 2026-02-22T21:14:18.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-22T21:14:18.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T21:14:18.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T21:14:18.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T21:14:18.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/1 2026-02-22T21:14:18.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/1 --op list foo 2026-02-22T21:14:18.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 1 2026-02-22T21:14:18.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-22T21:14:18.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T21:14:18.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T21:14:18.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T21:14:18.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/1 2026-02-22T21:14:18.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-22T21:14:18.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:14:18.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:14:18.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:14:18.300 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2026-02-22T21:14:18.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2026-02-22T21:14:18.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:14:18.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T21:14:18.301 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-22T21:14:18.301 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T21:14:18.301 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:14:18.301 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:14:18.301 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:14:18.301 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:18.301 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:18.303 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:18.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:18.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:14:18.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-22T21:14:18.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-22T21:14:18.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:14:18.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:14:18.303 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:14:18.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:14:18.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T21:14:18.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T21:14:18.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/1 2026-02-22T21:14:18.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T21:14:18.304 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T21:14:18.304 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:14:18.305 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/1/whoami 2026-02-22T21:14:18.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T21:14:18.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T21:14:18.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T21:14:18.305 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T21:14:18.320 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:18.311+0000 7f81f819ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:18.321 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:18.315+0000 7f81f819ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:18.322 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:18.315+0000 7f81f819ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:18.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T21:14:18.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:14:18.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T21:14:18.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:14:18.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:14:18.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:18.462 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:14:18.462 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T21:14:18.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:18.463 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:14:18.620 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:14:18.758 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:18.751+0000 7f81f819ca40 -1 Falling back to public interface 2026-02-22T21:14:19.412 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:19.403+0000 7f81f819ca40 -1 osd.1 19 log_to_monitors true 2026-02-22T21:14:19.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:14:19.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:19.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:14:19.621 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:14:19.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:19.621 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:14:19.774 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [10,19) [v2:127.0.0.1:6810/2601176493,v1:127.0.0.1:6811/2601176493] [v2:127.0.0.1:6812/2601176493,v1:127.0.0.1:6813/2601176493] exists,up b649b842-0a09-47b4-a847-12ce2772be45 2026-02-22T21:14:19.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:14:19.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:14:19.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:14:19.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T21:14:19.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:14:19.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:14:19.774 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:14:19.774 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:14:19.775 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:14:19.775 INFO:tasks.workunit.client.0.trial150.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:14:19.775 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:14:19.775 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:14:19.775 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:14:19.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:14:19.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:14:19.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:14:19.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:14:19.875 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T21:14:19.876 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:14:20.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:14:20.031 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:14:20.031 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:14:20.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:14:20.031 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:20.031 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:14:20.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-22T21:14:20.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-22T21:14:20.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486' 2026-02-22T21:14:20.103 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:20.103 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:14:20.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=98784247810 2026-02-22T21:14:20.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 98784247810 2026-02-22T21:14:20.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-98784247810' 2026-02-22T21:14:20.174 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:20.174 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:14:20.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509444 2026-02-22T21:14:20.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509444 2026-02-22T21:14:20.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836486 1-98784247810 2-64424509444' 2026-02-22T21:14:20.246 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:20.246 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-22T21:14:20.246 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:20.247 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:14:20.248 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836486 2026-02-22T21:14:20.248 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:20.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836486 2026-02-22T21:14:20.250 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-22T21:14:20.250 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836486 2026-02-22T21:14:20.250 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:20.417 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-22T21:14:20.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:14:21.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:14:21.419 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:21.576 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836485 -lt 21474836486 2026-02-22T21:14:21.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:14:22.577 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T21:14:22.578 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:22.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-22T21:14:22.742 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:22.743 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-98784247810 2026-02-22T21:14:22.743 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:22.744 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:14:22.744 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-98784247810 2026-02-22T21:14:22.744 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:22.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=98784247810 2026-02-22T21:14:22.746 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 98784247810' 2026-02-22T21:14:22.746 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 98784247810 2026-02-22T21:14:22.746 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:14:22.917 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 98784247810 -lt 98784247810 2026-02-22T21:14:22.917 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:22.918 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509444 2026-02-22T21:14:22.918 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:22.919 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:14:22.919 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509444 2026-02-22T21:14:22.920 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:22.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509444 2026-02-22T21:14:22.921 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2026-02-22T21:14:22.921 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509444 2026-02-22T21:14:22.922 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:14:23.077 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509444 -lt 64424509444 2026-02-22T21:14:23.078 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:14:23.078 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:23.078 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:23.265 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:14:23.265 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:14:23.266 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:14:23.266 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:14:23.266 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:14:23.266 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:14:23.266 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:14:23.266 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:14:23.418 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T21:14:23.419 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:14:23.419 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:23.419 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:23.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T21:14:23.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-22T21:14:23.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T21:14:23.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-22T21:14:23.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T21:14:23.612 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T21:14:23.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T21:14:23.713 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:14:23.713 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:14:23.714 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:14:23.714 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:14:23.714 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:14:23.714 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:14:23.714 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:14:23.872 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T21:14:23.873 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:14:23.873 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:23.873 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:24.062 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T21:14:24.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T21:14:24.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T21:14:24.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T21:14:24.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T21:14:24.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T21:14:24.063 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T21:14:24.063 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T21:14:24.063 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T21:14:24.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T21:14:24.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T21:14:24.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T21:14:24.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T21:14:24.253 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T21:14:24.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T21:14:24.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:14:24.454 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:14:24.454 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:14:24.454 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:14:24.454 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:14:24.454 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:14:24.454 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:14:24.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T21:14:24.615 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:14:24.615 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:24.615 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:24.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T21:14:24.804 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T21:14:24.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T21:14:24.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T21:14:24.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T21:14:24.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T21:14:24.805 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T21:14:24.805 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T21:14:24.806 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T21:14:24.992 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T21:14:24.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T21:14:24.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 2 >= 13 )) 2026-02-22T21:14:24.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T21:14:24.993 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.4 2026-02-22T21:14:25.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T21:14:25.394 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:14:25.395 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:14:25.395 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:14:25.395 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:14:25.395 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:14:25.396 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:14:25.396 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:14:25.550 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T21:14:25.550 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:14:25.551 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:25.551 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:25.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T21:14:25.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T21:14:25.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T21:14:25.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T21:14:25.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T21:14:25.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T21:14:25.740 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T21:14:25.740 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T21:14:25.740 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T21:14:25.932 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T21:14:25.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T21:14:25.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 3 >= 13 )) 2026-02-22T21:14:25.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T21:14:25.933 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.8 2026-02-22T21:14:26.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T21:14:26.735 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:14:26.735 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:14:26.735 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:14:26.735 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:14:26.736 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:14:26.736 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:14:26.736 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:14:26.893 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:14:26.893 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:14:26.893 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:26.893 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:27.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:14:27.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:14:27.089 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:14:27.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:57: TEST_recover_unexpected: JSON='["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T21:14:27.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:58: TEST_recover_unexpected: echo 'JSON is ["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' 2026-02-22T21:14:27.090 INFO:tasks.workunit.client.0.trial150.stdout:JSON is ["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}] 2026-02-22T21:14:27.090 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:59: TEST_recover_unexpected: rm -f td/osd-unexpected-clone/_ td/osd-unexpected-clone/data 2026-02-22T21:14:27.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:60: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2026-02-22T21:14:27.091 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=1 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=1 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.1 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:14:27.092 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:14:27.093 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:14:27.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:14:27.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2026-02-22T21:14:27.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-22T21:14:27.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T21:14:27.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2026-02-22T21:14:27.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T21:14:27.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/1 2026-02-22T21:14:27.401 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/1 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-attr _ 2026-02-22T21:14:27.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 1 2026-02-22T21:14:27.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-22T21:14:27.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T21:14:27.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2026-02-22T21:14:27.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T21:14:27.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/1 2026-02-22T21:14:27.552 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-22T21:14:27.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:14:27.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:14:27.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:14:27.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/1' 2026-02-22T21:14:27.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/1/journal' 2026-02-22T21:14:27.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:14:27.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T21:14:27.553 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-22T21:14:27.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T21:14:27.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:14:27.553 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:14:27.553 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/1 2026-02-22T21:14:27.554 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2026-02-22T21:14:27.555 INFO:tasks.workunit.client.0.trial150.stderr:start osd.1 2026-02-22T21:14:27.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/1 --osd-journal=td/osd-unexpected-clone/1/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:14:27.555 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/1/whoami 2026-02-22T21:14:27.555 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2026-02-22T21:14:27.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T21:14:27.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T21:14:27.556 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T21:14:27.572 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:27.563+0000 7f656a803a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:27.573 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:27.567+0000 7f656a803a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:27.574 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:27.567+0000 7f656a803a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:27.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2026-02-22T21:14:27.718 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:14:27.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-22T21:14:27.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:14:27.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:14:27.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:27.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:14:27.719 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T21:14:27.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:27.719 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:14:27.877 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:14:27.950 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:27.943+0000 7f656a803a40 -1 Falling back to public interface 2026-02-22T21:14:28.582 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:28.575+0000 7f656a803a40 -1 osd.1 24 log_to_monitors true 2026-02-22T21:14:28.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:14:28.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:28.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:14:28.879 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:14:28.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:28.879 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-22T21:14:29.036 INFO:tasks.workunit.client.0.trial150.stderr:osd.1 up in weight 1 up_from 28 up_thru 28 down_at 25 last_clean_interval [23,24) [v2:127.0.0.1:6810/3825845919,v1:127.0.0.1:6811/3825845919] [v2:127.0.0.1:6812/3825845919,v1:127.0.0.1:6813/3825845919] exists,up b649b842-0a09-47b4-a847-12ce2772be45 2026-02-22T21:14:29.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:14:29.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:14:29.036 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:14:29.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T21:14:29.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:14:29.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:14:29.037 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:14:29.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:14:29.037 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:14:29.037 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:14:29.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:14:29.037 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:14:29.038 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:14:29.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:14:29.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:14:29.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:14:29.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:14:29.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T21:14:29.127 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:14:29.297 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:14:29.297 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:14:29.297 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:14:29.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:14:29.298 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:29.298 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:14:29.369 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-22T21:14:29.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-22T21:14:29.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488' 2026-02-22T21:14:29.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:29.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:14:29.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084290 2026-02-22T21:14:29.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084290 2026-02-22T21:14:29.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-120259084290' 2026-02-22T21:14:29.442 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:29.442 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:14:29.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509447 2026-02-22T21:14:29.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509447 2026-02-22T21:14:29.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836488 1-120259084290 2-64424509447' 2026-02-22T21:14:29.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:29.518 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-22T21:14:29.518 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:29.518 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:14:29.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836488 2026-02-22T21:14:29.519 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:29.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836488 2026-02-22T21:14:29.521 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-22T21:14:29.521 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836488 2026-02-22T21:14:29.521 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:29.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836487 -lt 21474836488 2026-02-22T21:14:29.678 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:14:30.679 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:14:30.679 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:30.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836488 2026-02-22T21:14:30.832 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:30.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084290 2026-02-22T21:14:30.833 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:30.834 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:14:30.834 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084290 2026-02-22T21:14:30.835 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:30.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084290 2026-02-22T21:14:30.836 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084290' 2026-02-22T21:14:30.836 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 120259084290 2026-02-22T21:14:30.836 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:14:31.005 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084290 -lt 120259084290 2026-02-22T21:14:31.006 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:31.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-64424509447 2026-02-22T21:14:31.006 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:31.007 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:14:31.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-64424509447 2026-02-22T21:14:31.008 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:31.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=64424509447 2026-02-22T21:14:31.009 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2026-02-22T21:14:31.009 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 64424509447 2026-02-22T21:14:31.009 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:14:31.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 64424509447 -lt 64424509447 2026-02-22T21:14:31.177 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:14:31.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:31.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:31.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:14:31.370 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:14:31.370 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:14:31.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:14:31.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:14:31.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:14:31.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:14:31.371 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:14:31.525 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:14:31.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:14:31.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:31.525 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:31.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:14:31.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:14:31.716 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:61: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:14:31.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:14:32.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:14:32.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2026-02-22T21:14:32.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-22T21:14:32.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T21:14:32.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T21:14:32.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T21:14:32.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2026-02-22T21:14:32.024 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' get-bytes td/osd-unexpected-clone/data 2026-02-22T21:14:32.186 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2026-02-22T21:14:32.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-22T21:14:32.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T21:14:32.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T21:14:32.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T21:14:32.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/2 2026-02-22T21:14:32.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-22T21:14:32.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:14:32.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:14:32.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:14:32.187 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2026-02-22T21:14:32.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2026-02-22T21:14:32.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:14:32.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T21:14:32.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-22T21:14:32.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T21:14:32.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:14:32.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:14:32.188 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:14:32.188 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:32.188 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:32.188 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:32.188 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:32.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:14:32.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-22T21:14:32.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-22T21:14:32.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:14:32.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:14:32.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:14:32.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:14:32.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T21:14:32.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T21:14:32.189 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/2 2026-02-22T21:14:32.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T21:14:32.190 INFO:tasks.workunit.client.0.trial150.stderr:start osd.2 2026-02-22T21:14:32.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:14:32.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/2/whoami 2026-02-22T21:14:32.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T21:14:32.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T21:14:32.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T21:14:32.191 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T21:14:32.205 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:32.199+0000 7fb0c9379a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:32.206 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:32.199+0000 7fb0c9379a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:32.207 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:32.199+0000 7fb0c9379a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:32.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T21:14:32.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:14:32.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T21:14:32.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:14:32.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:14:32.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:32.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:14:32.344 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T21:14:32.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:32.344 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:14:32.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:14:32.586 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:32.579+0000 7fb0c9379a40 -1 Falling back to public interface 2026-02-22T21:14:33.176 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:33.167+0000 7fb0c9379a40 -1 osd.2 29 log_to_monitors true 2026-02-22T21:14:33.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:14:33.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:33.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:14:33.508 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:14:33.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:33.508 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:14:33.667 INFO:tasks.workunit.client.0.trial150.stderr:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 30 last_clean_interval [15,29) [v2:127.0.0.1:6818/379060572,v1:127.0.0.1:6819/379060572] [v2:127.0.0.1:6820/379060572,v1:127.0.0.1:6821/379060572] exists,up d0b51f2f-9685-4c8e-86d4-49fad0ce131a 2026-02-22T21:14:33.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:14:33.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:14:33.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:14:33.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T21:14:33.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:14:33.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:14:33.667 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:14:33.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:14:33.668 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:14:33.668 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:14:33.668 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:14:33.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:14:33.669 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:14:33.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:14:33.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:14:33.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:14:33.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:14:33.768 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T21:14:33.769 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:14:33.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:14:33.926 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:14:33.926 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:14:33.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:14:33.926 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:33.926 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:14:33.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-22T21:14:33.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-22T21:14:33.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490' 2026-02-22T21:14:33.999 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:33.999 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:14:34.073 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084292 2026-02-22T21:14:34.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084292 2026-02-22T21:14:34.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-120259084292' 2026-02-22T21:14:34.074 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:34.074 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:14:34.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=141733920770 2026-02-22T21:14:34.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 141733920770 2026-02-22T21:14:34.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836490 1-120259084292 2-141733920770' 2026-02-22T21:14:34.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:34.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-22T21:14:34.146 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:34.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:14:34.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836490 2026-02-22T21:14:34.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:34.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836490 2026-02-22T21:14:34.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-22T21:14:34.149 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836490 2026-02-22T21:14:34.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:34.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-22T21:14:34.311 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:14:35.312 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:14:35.312 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:35.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836489 -lt 21474836490 2026-02-22T21:14:35.474 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:14:36.475 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T21:14:36.475 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:36.635 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836491 -lt 21474836490 2026-02-22T21:14:36.636 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:36.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084292 2026-02-22T21:14:36.636 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:36.637 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:14:36.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084292 2026-02-22T21:14:36.637 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:36.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084292 2026-02-22T21:14:36.639 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084292' 2026-02-22T21:14:36.639 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 120259084292 2026-02-22T21:14:36.639 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:14:36.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084292 -lt 120259084292 2026-02-22T21:14:36.808 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:36.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-141733920770 2026-02-22T21:14:36.809 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:36.809 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:14:36.810 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-141733920770 2026-02-22T21:14:36.810 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:36.811 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=141733920770 2026-02-22T21:14:36.812 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 141733920770' 2026-02-22T21:14:36.812 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 141733920770 2026-02-22T21:14:36.812 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:14:36.970 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 141733920770 -lt 141733920770 2026-02-22T21:14:36.970 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:14:36.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:36.971 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:37.161 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:14:37.162 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:14:37.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:14:37.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:14:37.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:14:37.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:14:37.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:14:37.162 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:14:37.317 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:14:37.318 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:14:37.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:37.319 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:37.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:14:37.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:14:37.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:14:37.506 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:63: TEST_recover_unexpected: rados -p foo rmsnap snap 2026-02-22T21:14:37.573 INFO:tasks.workunit.client.0.trial150.stdout:removed pool foo snap snap 2026-02-22T21:14:37.575 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:65: TEST_recover_unexpected: sleep 5 2026-02-22T21:14:42.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:67: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2026-02-22T21:14:42.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-22T21:14:42.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T21:14:42.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T21:14:42.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T21:14:42.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2026-02-22T21:14:42.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-22T21:14:42.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T21:14:42.578 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T21:14:42.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T21:14:42.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2026-02-22T21:14:42.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:14:42.579 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:14:42.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:14:42.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:14:42.579 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:14:42.887 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:14:42.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2026-02-22T21:14:42.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-22T21:14:42.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T21:14:42.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T21:14:42.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T21:14:42.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2026-02-22T21:14:42.888 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-bytes td/osd-unexpected-clone/data 2026-02-22T21:14:43.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2026-02-22T21:14:43.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-22T21:14:43.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T21:14:43.174 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T21:14:43.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T21:14:43.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/2 2026-02-22T21:14:43.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-22T21:14:43.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:14:43.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:14:43.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:14:43.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2026-02-22T21:14:43.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2026-02-22T21:14:43.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:14:43.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T21:14:43.175 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-22T21:14:43.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T21:14:43.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:14:43.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:14:43.176 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:14:43.176 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:43.176 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:43.176 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:43.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:43.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:14:43.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-22T21:14:43.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-22T21:14:43.176 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:14:43.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:14:43.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:14:43.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:14:43.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T21:14:43.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T21:14:43.177 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/2 2026-02-22T21:14:43.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T21:14:43.178 INFO:tasks.workunit.client.0.trial150.stderr:start osd.2 2026-02-22T21:14:43.178 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:14:43.178 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/2/whoami 2026-02-22T21:14:43.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T21:14:43.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T21:14:43.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T21:14:43.179 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T21:14:43.193 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:43.187+0000 7f09998baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:43.195 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:43.187+0000 7f09998baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:43.196 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:43.187+0000 7f09998baa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:43.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T21:14:43.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:14:43.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T21:14:43.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:14:43.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:14:43.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:43.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:14:43.336 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T21:14:43.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:43.336 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:14:43.493 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:14:43.590 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:43.583+0000 7f09998baa40 -1 Falling back to public interface 2026-02-22T21:14:44.132 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:44.123+0000 7f09998baa40 -1 osd.2 35 log_to_monitors true 2026-02-22T21:14:44.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:14:44.494 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:44.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:14:44.495 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:14:44.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:44.495 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:14:44.650 INFO:tasks.workunit.client.0.trial150.stderr:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 36 last_clean_interval [33,35) [v2:127.0.0.1:6818/3689660036,v1:127.0.0.1:6819/3689660036] [v2:127.0.0.1:6820/3689660036,v1:127.0.0.1:6821/3689660036] exists,up d0b51f2f-9685-4c8e-86d4-49fad0ce131a 2026-02-22T21:14:44.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:14:44.650 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:14:44.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:14:44.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T21:14:44.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:14:44.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:14:44.651 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:14:44.651 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:14:44.651 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:14:44.651 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:14:44.651 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:14:44.651 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:14:44.652 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:14:44.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:14:44.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:14:44.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:14:44.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:14:44.749 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T21:14:44.749 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:14:44.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:14:44.910 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:14:44.910 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:14:44.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:14:44.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:44.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:14:44.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836494 2026-02-22T21:14:44.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836494 2026-02-22T21:14:44.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836494' 2026-02-22T21:14:44.982 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:44.982 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:14:45.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084295 2026-02-22T21:14:45.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084295 2026-02-22T21:14:45.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836494 1-120259084295' 2026-02-22T21:14:45.053 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:45.053 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:14:45.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=167503724546 2026-02-22T21:14:45.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 167503724546 2026-02-22T21:14:45.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836494 1-120259084295 2-167503724546' 2026-02-22T21:14:45.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:45.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836494 2026-02-22T21:14:45.128 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:45.129 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:14:45.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836494 2026-02-22T21:14:45.130 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:45.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836494 2026-02-22T21:14:45.131 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2026-02-22T21:14:45.131 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836494 2026-02-22T21:14:45.132 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:45.286 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836492 -lt 21474836494 2026-02-22T21:14:45.287 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:14:46.288 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:14:46.288 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:46.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836494 -lt 21474836494 2026-02-22T21:14:46.449 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:46.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084295 2026-02-22T21:14:46.450 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:46.451 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:14:46.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084295 2026-02-22T21:14:46.451 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:46.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084295 2026-02-22T21:14:46.453 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084295' 2026-02-22T21:14:46.453 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 120259084295 2026-02-22T21:14:46.453 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:14:46.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084295 -lt 120259084295 2026-02-22T21:14:46.614 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:46.614 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-167503724546 2026-02-22T21:14:46.615 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:46.616 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:14:46.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-167503724546 2026-02-22T21:14:46.616 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:46.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=167503724546 2026-02-22T21:14:46.618 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 167503724546' 2026-02-22T21:14:46.618 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 167503724546 2026-02-22T21:14:46.618 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:14:46.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 167503724546 -lt 167503724546 2026-02-22T21:14:46.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:14:46.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:46.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:46.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:14:46.980 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:14:46.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:14:46.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:14:46.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:14:46.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:14:46.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:14:46.980 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:14:47.139 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:14:47.139 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:14:47.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:47.140 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:47.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:14:47.333 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:68: TEST_recover_unexpected: objectstore_tool td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/osd-unexpected-clone 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=2 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/osd-unexpected-clone 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=2 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/osd-unexpected-clone TERM osd.2 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:14:47.334 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:14:47.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:14:47.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-unexpected-clone 2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2026-02-22T21:14:47.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-unexpected-clone 2026-02-22T21:14:47.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-22T21:14:47.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2026-02-22T21:14:47.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-22T21:14:47.642 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-unexpected-clone/2 2026-02-22T21:14:47.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-unexpected-clone/2 '["1.0",{"oid":"foo","key":"","snapid":1,"hash":2143417350,"max":0,"pool":1,"namespace":"","max":0}]' set-attr _ td/osd-unexpected-clone/_ 2026-02-22T21:14:47.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/osd-unexpected-clone 2 2026-02-22T21:14:47.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-unexpected-clone 2026-02-22T21:14:47.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-22T21:14:47.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2026-02-22T21:14:47.908 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-22T21:14:47.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-unexpected-clone/2 2026-02-22T21:14:47.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 ' 2026-02-22T21:14:47.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-22T21:14:47.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-22T21:14:47.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-22T21:14:47.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-unexpected-clone/2' 2026-02-22T21:14:47.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-unexpected-clone/2/journal' 2026-02-22T21:14:47.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-22T21:14:47.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-22T21:14:47.909 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-unexpected-clone' 2026-02-22T21:14:47.909 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-22T21:14:47.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-22T21:14:47.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-22T21:14:47.910 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-22T21:14:47.910 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:14:47.910 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:14:47.910 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:47.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' 2026-02-22T21:14:47.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-22T21:14:47.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-unexpected-clone/$name.log' 2026-02-22T21:14:47.910 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-unexpected-clone/$name.pid' 2026-02-22T21:14:47.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-22T21:14:47.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-22T21:14:47.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-22T21:14:47.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-22T21:14:47.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-22T21:14:47.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-22T21:14:47.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-unexpected-clone/2 2026-02-22T21:14:47.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2026-02-22T21:14:47.911 INFO:tasks.workunit.client.0.trial150.stderr:start osd.2 2026-02-22T21:14:47.911 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=b572a56e-e1e8-439a-be77-6560ea29a62d --auth-supported=none --mon-host=127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-unexpected-clone/2 --osd-journal=td/osd-unexpected-clone/2/journal --chdir= --run-dir=td/osd-unexpected-clone '--admin-socket=/tmp/ceph-asok.662307/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-unexpected-clone/$name.log' '--pid-file=td/osd-unexpected-clone/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-22T21:14:47.911 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-unexpected-clone/2/whoami 2026-02-22T21:14:47.912 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2026-02-22T21:14:47.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-22T21:14:47.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-22T21:14:47.913 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-22T21:14:47.924 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:47.919+0000 7f4d82475a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:47.926 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:47.919+0000 7f4d82475a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:47.927 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:47.919+0000 7f4d82475a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-22T21:14:48.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2026-02-22T21:14:48.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-22T21:14:48.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-22T21:14:48.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-22T21:14:48.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-22T21:14:48.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:48.068 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-22T21:14:48.068 INFO:tasks.workunit.client.0.trial150.stderr:0 2026-02-22T21:14:48.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:48.069 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:14:48.225 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-22T21:14:48.294 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:48.287+0000 7f4d82475a40 -1 Falling back to public interface 2026-02-22T21:14:48.907 INFO:tasks.workunit.client.0.trial150.stderr:2026-02-22T21:14:48.899+0000 7f4d82475a40 -1 osd.2 41 log_to_monitors true 2026-02-22T21:14:49.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-22T21:14:49.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-22T21:14:49.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-22T21:14:49.226 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:14:49.226 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-22T21:14:49.227 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-22T21:14:49.382 INFO:tasks.workunit.client.0.trial150.stderr:osd.2 up in weight 1 up_from 45 up_thru 0 down_at 42 last_clean_interval [39,41) [v2:127.0.0.1:6818/65969967,v1:127.0.0.1:6819/65969967] [v2:127.0.0.1:6820/65969967,v1:127.0.0.1:6821/65969967] exists,up d0b51f2f-9685-4c8e-86d4-49fad0ce131a 2026-02-22T21:14:49.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-22T21:14:49.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-22T21:14:49.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-22T21:14:49.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-22T21:14:49.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-22T21:14:49.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-22T21:14:49.382 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-22T21:14:49.382 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-22T21:14:49.383 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-22T21:14:49.384 INFO:tasks.workunit.client.0.trial150.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-22T21:14:49.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-22T21:14:49.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-22T21:14:49.384 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-22T21:14:49.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-22T21:14:49.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-22T21:14:49.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-22T21:14:49.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-22T21:14:49.485 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-22T21:14:49.485 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-22T21:14:49.643 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-22T21:14:49.643 INFO:tasks.workunit.client.0.trial150.stderr:1 2026-02-22T21:14:49.644 INFO:tasks.workunit.client.0.trial150.stderr:2' 2026-02-22T21:14:49.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-22T21:14:49.644 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:49.644 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2026-02-22T21:14:49.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836496 2026-02-22T21:14:49.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836496 2026-02-22T21:14:49.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496' 2026-02-22T21:14:49.717 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:49.717 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2026-02-22T21:14:49.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=120259084297 2026-02-22T21:14:49.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 120259084297 2026-02-22T21:14:49.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-120259084297' 2026-02-22T21:14:49.788 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-22T21:14:49.788 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2026-02-22T21:14:49.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=193273528322 2026-02-22T21:14:49.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 193273528322 2026-02-22T21:14:49.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: flush_pg_stats: seqs=' 0-21474836496 1-120259084297 2-193273528322' 2026-02-22T21:14:49.859 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:49.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836496 2026-02-22T21:14:49.859 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:49.860 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=0 2026-02-22T21:14:49.860 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 0-21474836496 2026-02-22T21:14:49.861 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:49.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=21474836496 2026-02-22T21:14:49.862 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-22T21:14:49.862 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.0 seq 21474836496 2026-02-22T21:14:49.863 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-22T21:14:50.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 21474836496 -lt 21474836496 2026-02-22T21:14:50.027 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:50.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-120259084297 2026-02-22T21:14:50.028 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:50.029 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=1 2026-02-22T21:14:50.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 1-120259084297 2026-02-22T21:14:50.029 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:50.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=120259084297 2026-02-22T21:14:50.030 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.1 seq 120259084297' 2026-02-22T21:14:50.030 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.1 seq 120259084297 2026-02-22T21:14:50.030 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-22T21:14:50.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 120259084297 -lt 120259084297 2026-02-22T21:14:50.190 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: for s in $seqs 2026-02-22T21:14:50.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-193273528322 2026-02-22T21:14:50.191 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 1 2026-02-22T21:14:50.192 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: osd=2 2026-02-22T21:14:50.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 2-193273528322 2026-02-22T21:14:50.192 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: cut -d - -f 2 2026-02-22T21:14:50.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: seq=193273528322 2026-02-22T21:14:50.193 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: echo 'waiting osd.2 seq 193273528322' 2026-02-22T21:14:50.193 INFO:tasks.workunit.client.0.trial150.stderr:waiting osd.2 seq 193273528322 2026-02-22T21:14:50.193 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:14:50.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 193273528322 2026-02-22T21:14:50.352 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:14:51.353 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-22T21:14:51.354 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:14:51.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 1 -lt 193273528322 2026-02-22T21:14:51.516 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: sleep 1 2026-02-22T21:14:52.517 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-22T21:14:52.518 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-22T21:14:52.677 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: test 193273528322 -lt 193273528322 2026-02-22T21:14:52.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-22T21:14:52.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:52.678 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:52.872 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 1 == 0 2026-02-22T21:14:52.873 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:14:52.873 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:14:52.873 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:14:52.873 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:14:52.873 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:14:52.873 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:14:52.873 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:14:53.033 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T21:14:53.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:14:53.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:53.034 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:53.233 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T21:14:53.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' -1 2026-02-22T21:14:53.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: loop=0 2026-02-22T21:14:53.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: num_active_clean=0 2026-02-22T21:14:53.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T21:14:53.234 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.1 2026-02-22T21:14:53.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T21:14:53.335 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:14:53.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:14:53.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:14:53.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:14:53.335 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:14:53.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:14:53.336 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:14:53.490 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=0 2026-02-22T21:14:53.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:14:53.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:53.490 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:53.683 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 0 = 1 2026-02-22T21:14:53.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: test 0 '!=' 0 2026-02-22T21:14:53.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1667: wait_for_clean: get_is_making_recovery_progress 2026-02-22T21:14:53.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-22T21:14:53.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-22T21:14:53.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-22T21:14:53.684 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-22T21:14:53.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-22T21:14:53.684 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-22T21:14:53.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-22T21:14:53.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-22T21:14:53.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: (( 1 >= 13 )) 2026-02-22T21:14:53.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1674: wait_for_clean: eval 2026-02-22T21:14:53.878 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1675: wait_for_clean: sleep 0.2 2026-02-22T21:14:54.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1676: wait_for_clean: loop+=1 2026-02-22T21:14:54.078 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-22T21:14:54.078 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-22T21:14:54.078 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-22T21:14:54.078 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-22T21:14:54.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-22T21:14:54.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-22T21:14:54.079 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-22T21:14:54.241 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=1 2026-02-22T21:14:54.241 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-22T21:14:54.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-22T21:14:54.242 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-22T21:14:54.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 1 = 1 2026-02-22T21:14:54.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-22T21:14:54.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-22T21:14:54.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:70: TEST_recover_unexpected: sleep 5 2026-02-22T21:14:59.437 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:72: TEST_recover_unexpected: ceph pg repair 1.0 2026-02-22T21:14:59.586 INFO:tasks.workunit.client.0.trial150.stderr:instructing pg 1.0 on osd.1 to repair 2026-02-22T21:14:59.596 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:74: TEST_recover_unexpected: sleep 10 2026-02-22T21:15:09.597 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:76: TEST_recover_unexpected: ceph log last 2026-02-22T21:15:09.778 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:00.552885+0000 osd.1 (osd.1) 2 : cluster [ERR] 1.0 shard 0 1:602f83fe:::foo:1 : missing 2026-02-22T21:15:09.778 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:00.552887+0000 osd.1 (osd.1) 3 : cluster [ERR] 1.0 shard 1 1:602f83fe:::foo:1 : missing 2026-02-22T21:15:09.778 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:00.552979+0000 osd.1 (osd.1) 4 : cluster [ERR] repair 1.0 1:602f83fe:::foo:1 : is an unexpected clone 2026-02-22T21:15:09.778 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:00.553090+0000 osd.1 (osd.1) 5 : cluster [ERR] 1.0 repair 1 missing, 0 inconsistent objects 2026-02-22T21:15:09.779 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:00.553105+0000 osd.1 (osd.1) 6 : cluster [ERR] 1.0 repair 1 missing, 0 inconsistent objects 2026-02-22T21:15:09.779 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:00.553129+0000 osd.1 (osd.1) 7 : cluster [ERR] 1.0 repair 3 errors, 2 fixed 2026-02-22T21:15:09.779 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:01.536623+0000 osd.1 (osd.1) 9 : cluster [ERR] 1.0 shard 1 soid 1:602f83fe:::foo:1 : candidate size 0 info size 15900 mismatch 2026-02-22T21:15:09.779 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:01.536686+0000 osd.1 (osd.1) 10 : cluster [ERR] 1.0 shard 0 soid 1:602f83fe:::foo:1 : data_digest 0x9f81759d != data_digest 0x5af226ef from auth oi 1:602f83fe:::foo:1(19'2 client.4156.0:1 dirty|data_digest s 15900 uv 1 dd 5af226ef alloc_hint [0 0 0]) 2026-02-22T21:15:09.779 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:01.536688+0000 osd.1 (osd.1) 11 : cluster [ERR] 1.0 shard 1 soid 1:602f83fe:::foo:1 : data_digest 0xffffffff != data_digest 0x9f81759d from shard 0, data_digest 0xffffffff != data_digest 0x5af226ef from auth oi 1:602f83fe:::foo:1(19'2 client.4156.0:1 dirty|data_digest s 15900 uv 1 dd 5af226ef alloc_hint [0 0 0]), size 0 != size 15900 from auth oi 1:602f83fe:::foo:1(19'2 client.4156.0:1 dirty|data_digest s 15900 uv 1 dd 5af226ef alloc_hint [0 0 0]), size 0 != size 15900 from shard 0 2026-02-22T21:15:09.779 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:01.536697+0000 osd.1 (osd.1) 12 : cluster [ERR] 1.0 soid 1:602f83fe:::foo:1 : data_digest 0x5af226ef != data_digest 0x9f81759d from shard 0 2026-02-22T21:15:09.779 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:01.536784+0000 osd.1 (osd.1) 13 : cluster [ERR] deep-scrub 1.0 1:602f83fe:::foo:1 : is an unexpected clone 2026-02-22T21:15:09.779 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:01.536966+0000 osd.1 (osd.1) 14 : cluster [ERR] 1.0 deep-scrub 0 missing, 1 inconsistent objects 2026-02-22T21:15:09.779 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:01.536968+0000 osd.1 (osd.1) 15 : cluster [ERR] 1.0 deep-scrub 4 errors 2026-02-22T21:15:09.779 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:03.851591+0000 mon.a (mon.0) 255 : cluster [ERR] Health check failed: 4 scrub errors (OSD_SCRUB_ERRORS) 2026-02-22T21:15:09.779 INFO:tasks.workunit.client.0.trial150.stdout:2026-02-22T21:15:03.851613+0000 mon.a (mon.0) 256 : cluster [ERR] Health check failed: Possible data damage: 1 pg inconsistent (PG_DAMAGED) 2026-02-22T21:15:09.787 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:79: TEST_recover_unexpected: timeout 60 ceph tell osd.0 version 2026-02-22T21:15:09.853 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:15:09.854 INFO:tasks.workunit.client.0.trial150.stdout: "version": "20.2.0-794-ga5c3084f", 2026-02-22T21:15:09.854 INFO:tasks.workunit.client.0.trial150.stdout: "release": "tentacle", 2026-02-22T21:15:09.854 INFO:tasks.workunit.client.0.trial150.stdout: "release_type": "stable" 2026-02-22T21:15:09.854 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:15:09.861 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:80: TEST_recover_unexpected: timeout 60 ceph tell osd.1 version 2026-02-22T21:15:09.925 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:15:09.926 INFO:tasks.workunit.client.0.trial150.stdout: "version": "20.2.0-794-ga5c3084f", 2026-02-22T21:15:09.926 INFO:tasks.workunit.client.0.trial150.stdout: "release": "tentacle", 2026-02-22T21:15:09.926 INFO:tasks.workunit.client.0.trial150.stdout: "release_type": "stable" 2026-02-22T21:15:09.926 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:15:09.935 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:81: TEST_recover_unexpected: timeout 60 ceph tell osd.2 version 2026-02-22T21:15:10.001 INFO:tasks.workunit.client.0.trial150.stdout:{ 2026-02-22T21:15:10.001 INFO:tasks.workunit.client.0.trial150.stdout: "version": "20.2.0-794-ga5c3084f", 2026-02-22T21:15:10.001 INFO:tasks.workunit.client.0.trial150.stdout: "release": "tentacle", 2026-02-22T21:15:10.001 INFO:tasks.workunit.client.0.trial150.stdout: "release_type": "stable" 2026-02-22T21:15:10.001 INFO:tasks.workunit.client.0.trial150.stdout:} 2026-02-22T21:15:10.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/scrub/osd-unexpected-clone.sh:35: run: teardown td/osd-unexpected-clone 2026-02-22T21:15:10.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-unexpected-clone 2026-02-22T21:15:10.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-22T21:15:10.011 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-unexpected-clone KILL 2026-02-22T21:15:10.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:15:10.011 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:15:10.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:15:10.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:15:10.012 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:15:10.120 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:15:10.120 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:15:10.121 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:15:10.122 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:15:10.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:15:10.123 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:15:10.124 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:15:10.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:15:10.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:15:10.125 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:15:10.125 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:15:10.127 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:15:10.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-22T21:15:10.128 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-unexpected-clone 2026-02-22T21:15:10.142 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:15:10.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:15:10.143 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:15:10.143 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.662307 2026-02-22T21:15:10.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:15:10.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:15:10.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-22T21:15:10.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/osd-unexpected-clone 0 2026-02-22T21:15:10.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-unexpected-clone 2026-02-22T21:15:10.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-22T21:15:10.144 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-unexpected-clone KILL 2026-02-22T21:15:10.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-22T21:15:10.144 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-22T21:15:10.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-22T21:15:10.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-22T21:15:10.145 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-22T21:15:10.147 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-22T21:15:10.148 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-22T21:15:10.148 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-22T21:15:10.149 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-22T21:15:10.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-22T21:15:10.149 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-22T21:15:10.150 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-22T21:15:10.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:15:10.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-22T21:15:10.151 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-22T21:15:10.152 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-22T21:15:10.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:15:10.153 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-22T21:15:10.154 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-unexpected-clone 2026-02-22T21:15:10.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-22T21:15:10.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-22T21:15:10.155 INFO:tasks.workunit.client.0.trial150.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.662307 2026-02-22T21:15:10.155 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.662307 2026-02-22T21:15:10.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-22T21:15:10.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-22T21:15:10.156 INFO:tasks.workunit.client.0.trial150.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-22T21:15:10.157 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-22T21:15:10.157 DEBUG:teuthology.orchestra.run.trial150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-22T21:15:10.208 INFO:tasks.workunit:Stopping ['scrub'] on client.0... 2026-02-22T21:15:10.208 DEBUG:teuthology.orchestra.run.trial150:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-22T21:15:10.557 DEBUG:teuthology.parallel:result is None 2026-02-22T21:15:10.558 DEBUG:teuthology.orchestra.run.trial150:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-22T21:15:10.565 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-22T21:15:10.565 DEBUG:teuthology.orchestra.run.trial150:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-22T21:15:10.615 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-22T21:15:10.615 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-22T21:15:10.630 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-22T21:15:10.630 DEBUG:teuthology.orchestra.run.trial150:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-22T21:15:10.766 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-22T21:15:10.779 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-22T21:15:10.779 DEBUG:teuthology.orchestra.run.trial150:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-22T21:15:10.833 INFO:teuthology.orchestra.run.trial150.stdout: remote refid st t when poll reach delay offset jitter 2026-02-22T21:15:10.833 INFO:teuthology.orchestra.run.trial150.stdout:============================================================================== 2026-02-22T21:15:10.833 INFO:teuthology.orchestra.run.trial150.stdout:-172-233-177-198 80.192.165.246 2 u 11 128 377 36.896 +8.620 1.191 2026-02-22T21:15:10.833 INFO:teuthology.orchestra.run.trial150.stdout:*rn-02.koehn.com 17.253.6.37 2 u 124 128 377 19.513 +4.959 1.836 2026-02-22T21:15:10.833 INFO:teuthology.orchestra.run.trial150.stdout:+ns2.fortrockdc. 129.134.25.123 2 u 61 128 377 65.195 +1.954 4.095 2026-02-22T21:15:10.833 INFO:teuthology.orchestra.run.trial150.stdout:+us-east-2.clear 130.207.244.240 2 u 54 128 377 37.043 +4.577 0.432 2026-02-22T21:15:10.834 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-22T21:15:10.845 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-22T21:15:10.846 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-22T21:15:10.860 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-22T21:15:10.873 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-22T21:15:10.887 INFO:teuthology.task.internal:Duration was 3484.430742 seconds 2026-02-22T21:15:10.887 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-22T21:15:10.901 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-22T21:15:10.901 DEBUG:teuthology.orchestra.run.trial150:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-22T21:15:10.927 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-22T21:15:10.927 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial150.front.sepia.ceph.com 2026-02-22T21:15:10.927 DEBUG:teuthology.orchestra.run.trial150:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-22T21:15:10.974 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-22T21:15:10.974 DEBUG:teuthology.orchestra.run.trial150:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T21:15:11.237 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-22T21:15:11.237 DEBUG:teuthology.orchestra.run.trial150:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-22T21:15:11.247 INFO:teuthology.orchestra.run.trial150.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-22T21:15:11.248 INFO:teuthology.orchestra.run.trial150.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-22T21:15:11.248 INFO:teuthology.orchestra.run.trial150.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-22T21:15:11.248 INFO:teuthology.orchestra.run.trial150.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-22T21:15:11.248 INFO:teuthology.orchestra.run.trial150.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-22T21:15:11.315 INFO:teuthology.orchestra.run.trial150.stderr: 81.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-22T21:15:11.317 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-22T21:15:11.333 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-22T21:15:11.334 DEBUG:teuthology.orchestra.run.trial150:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-22T21:15:11.368 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-22T21:15:11.385 DEBUG:teuthology.orchestra.run.trial150:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:15:11.423 INFO:teuthology.orchestra.run.trial150.stdout:kernel.core_pattern = core 2026-02-22T21:15:11.432 DEBUG:teuthology.orchestra.run.trial150:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-22T21:15:11.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-22T21:15:11.479 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-22T21:15:11.491 INFO:teuthology.task.internal:Transferring archived files... 2026-02-22T21:15:11.493 DEBUG:teuthology.misc:Transferring archived files from trial150:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/nmordech-2026-02-22_19:44:01-rados-wip-rocky10-branch-of-the-day-2026-02-22-1771776513-tentacle-distro-default-trial/64480/remote/trial150 2026-02-22T21:15:11.494 DEBUG:teuthology.orchestra.run.trial150:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-22T21:15:11.553 INFO:teuthology.task.internal:Removing archive directory... 2026-02-22T21:15:11.554 DEBUG:teuthology.orchestra.run.trial150:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-22T21:15:11.579 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-22T21:15:11.594 INFO:teuthology.task.internal:Not uploading archives. 2026-02-22T21:15:11.595 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-22T21:15:11.606 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-22T21:15:11.606 DEBUG:teuthology.orchestra.run.trial150:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-22T21:15:11.624 INFO:teuthology.orchestra.run.trial150.stdout: 1179687 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 22 21:15 /home/ubuntu/cephtest 2026-02-22T21:15:11.625 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-22T21:15:11.641 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-22T21:15:11.677 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/scrub} duration: 3484.4307420253754 flavor: default owner: scheduled_nmordech@soko04.front.sepia.ceph.com success: true 2026-02-22T21:15:11.677 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-22T21:15:11.787 INFO:teuthology.run:pass